Talk

This talk will showcase how to leverage containers to solve UI-Test infrastructure for Android.

Is it reasonable or not to control the quality of a test code? Why do Software Engineers in Test have to follow the same rules and conventions as developers do? Is there any tool which may help to partially automate and simplify the code review process? Should it be integrated into main development flow?

It’s time to change the way you look at Test Automation. In this talk you’ll see practical examples of the use of code quality tools, and understand how they can be applied in the CI flow.

Over the years, I’ve learned one key lesson from my experience in automating test cases. Regardless of how much effort I put into making my test suites the most well-designed, high-quality work of software engineering, the fact remains that all these test cases are useless if I don’t have a test infrastructure to adequately support running them. In my experience test environment availability is very limited and I need to be able to work with minimal resources.

For example, on a recent project I was responsible for, I had a suite of REST API tests, which needed to be run continuously in a collection of environments that mirrored a variety of customer environment configurations (OS, database server, application server, browser).

I could have gone the traditional route of having a permanent environment in place for each environment configuration but this would have been very draining on the limited resources available to me. Also, in my case, my test suite was made up of API tests, which meant they executed relatively quickly (compared to say UI tests). To have multiple environments dedicated to running a short test cycle a few times a day seemed to me like a massive waste of resources.

To solve this problem, I looked into using Docker containers. This way I could configure, create and remove environments easily, quickly so as best to suit our own environmental requirements.

So in this talk, I would like to take the audience through my journey of how I used Docker and CI tools to overcome the challenges I had around lack of test infrastructure resources.

In this talk I want to share our own experience testing on prod. Our users help us test and customize our algorithms and do not even know about it!

How it works?

Let’s discuss…

Powerful tools give a lot room for improvements. In order to make it valuable for you, I define your way of configuration. This talk will represent use cases of ReportPortal and the most beneficial configurations of usage. Based on real project examples, with collected metrics and efficiency improvement for them.

Internet changed our lives. We no longer go to book store since there is Amazon. We no longer watch TV since there are YouTube and Netflix. We tend to migrate into Internet and this is generates huge amount of traffic. This brings us new challenges but old solutions fail to keep pace with changes.

During past few years microservice oriented design became standard for distributed high load systems. It’s been proven by lots of companies including FANG (Facebook, Amazon, Netflix and Google) so at this stage efficiency of this approach is indisputable. Unfortunately there is no silver bullet and with all good parts we’ve received operation complexity, configuration management complexity, difficulties with debugging and tracing, instability of interfaces and performance drawback caused by expenses of IPC.

Goal of this talk is to get through performance analysis process of simple calculator application implemented as a set of microservices with following FOSS toolset: k6 and gatling for traffic generation, prometheus for monitoring, perf Linux profiler and FlameGraph project for flame graphs generation.

The talk is about a possibility of creation non-static code analyzer which will point a dev team to some unobvious bugs in the system based on machine learning technics.

We will become acquainted with Weka — the machine learning tool for exploratory development of the algorithm, also, we will take a closer look on basic algorithms for classification: Decision Trees, SVM and Naive Bayes classifier.

Real life examples will show you how the development life cycle can be improved with AI technics.

Basic functionalities of a peer 2 peer communication application are text, voice messaging, VoIP and video calls. In order to test its functionalities, we need more than two devices to communicate with each other. There are also network interruptions / changes or app notification installed on mobile device which can harm the quality of communication and application.

In order to automate regression test of those kind of application, we have extended Appium and Selenium Grid where we are able to cross communicate “n” number devices with each other. We also created a layer manipulating the mobile devices’ Bluetooth, wireless and other features.

Right now, we are able to automate P2P appliications test with more than 100 devices varying from iOS to Android.

Browser tests are known to be the flakiest ones. This is partly because browser infrastructure is complicated to maintain. But the second reason is – main stream browser automation tools such as Selenium server are far from being efficient.

As your company grows – your browser automation infrastructure should easily adapt to increased loads and new requirements. Unfortunately this is not so easy to get really scalable Selenium cluster using only the main stream tools. But there are some new open-source tools and approaches that can be used to deliver really scalable Selenium cluster.

This master-class covers an emerging browser automation tool – Selenoid, a truly efficient replacement of the standard Selenium server. I will explain how it works and why it is so fast. I will show in details features like video recording, sending logs and statistics to centralized storage and how to easily visualize this data.

Every automated testing system evolves over time. Let me tell you a story about how we started testing CloudForms. First with standard libraries and tools, then gradually developing and improving our own tools. Python played an important role, one which gave us the opportunities and freedom on how to approach the code. We subsequently did two refactors of the system. In the first, we focused on abstracting the UI interactions out of the test. Second, we carefully continued to iterate with feedback from our successes and failures. Instead of intertwined spaghetti code, we designed a couple of stand-alone libraries that jointly power our testing system with focus on declarativity. One library represents the abstraction of the UI, inspired by good ideas from Page Objects, but doing things better. The second one takes care of navigating around the UI in an intuitive way. And there are other ones as well. So let’s go through this story together and watch how we leveraged our Selenium testing with Python.

We invited experienced speakers to share their vision on WebDriver/Selenium and ecosystem around it. In addition following topics will be discusssed:

– WebDriver current state and nearest future;
– tools and frameworks built on top of WebDriver;
– approaches and techniques in test automation using WebDriver.

If you have questions to discuss with experienced professionals it is right place and time to do it.

Almost all systems actively integrate with external services. Some of these external dependencies are legacy and slow, while some are modern and fast. Problems occur when we want to test our system in an isolated predictable environment. This is very important because in the opposite case, tests become flaky and unreliable causing the effectiveness of testing to decrease very quickly. Therefore, when testing is done with real external dependencies, the following factors could affect it: network issues, slow responses, unexpected failures, data corruption, access issues, etc. This tends to occur more often when external systems are legacy. Every programmer has to face legacy code day after day. It might be ugly, it might look scary, and it has even been known to make a grown man cry. Creating automation testing in cases like these can easily turn your testing into a nightmare.

The first thing that can help is writing stubs for external dependencies and using them for testing purposes. Another problem arises immediately: stubs fix external behaviour and nobody can guarantee that it will remain the same forever, especially if the service is under active development. Because of this, you need not only to have stubs written but also to regularly inspect changes in behaviour and adapt them. These activities significantly increase our development efforts in the testing area. Of course, such stubs should be able to work with legacy SOAP-based systems.

This presentation shows The Smart Stub as one of the solutions to simplify your life and make testing great again. Here are a few scenarios where The Smart Stub can be used. The Smart Stub can be used for local needs and on the CI pipeline as well, allowing for automation of the following activities.

– Running existing tests using The Smart Stub as a proxy to record all request and response pairs.
– Creating flexible matching rules for requests based on recorded data.
– Customizing responses with flexible data replacement rules to ensure they’re always up-to-date.
– Using The Smart Stub in replay mode when all matched requests are processed by The Smart Stub and actualized recorded response is returned. All non-matched requests are passed through and recorded for future usage.
– Recording request and response pairs could be scheduled for comparison to real service in order to detect changes in external behaviour as quickly as possible.

Have you ever wondered how to test a light switch? Can you imagine it being one of the biggest challenges in your career? There are nearly six billion IoT (Internet of Things) devices. By 2020, this number may triple. Undoubtedly, it is a rapidly developing area of IT market.

I will take the participants for a journey to the IoT world. It will be a talk about the challenges that any tester will face at some point. I will present the dangers and snares but also good practices and practical approach to E2E test automation for the IoT solutions.

The main question is: how to test application for end user among so much hardware equipment? Technical examples will be presented using Python languages and supported by physical devices.

Technologies like Squarespace, Salesforce, WordPress, or WIX are extremely popular for those who want to create a working website without necessary developer knowledge. In this talk, I would explore how Salesforce uses Page Object Model patterns to test its Communities platform which is used to develop websites for Salesforce users.

Throughout the talk, we will explore how multi-frame platform can be directly mapped to POM for Selenium Webdriver, and how client side code is developed to support this pattern.

The importance and complexity of these test framework is that it needs to be applicable for both platform and produced websites.

If you think about tests rather than corn flakes whenever you hear the term “Flaky”, this talk might be for you.

I’ll share some thoughts about the relationship between quality and culture, the battle of manual VS. automation, and the notion of test pyramid. In addition, I’ll describe how we bake quality into products and processes at Google.

With more breaking points and less stability, E2E tests are very hard to maintain. The automation guild at WixEngineering have reduced the amount of flaky tests by utilizing what we define as the “Test State Pattern”, where every test transitions between states according to its behavior.

We will talk about the motivation, implementation and discuss how everyone can achieve the same.

This talk is a brief overview of how we deal with UI test architecture on our project. Complex backend, SPA as a frontend, 20+ different brands and 100+ features for each, more and more code.

We will discuss:

– fakes and test doubles, approaches how to build and maintain them;
– development patterns, which help you to make your architecture more simple, stable and readable;
– tips and tricks – how to make your life with UI tests easier.

Not all desired capabilities where born equal:

– Some were born in the world of open source, while others were born as vendor proprietary properties.
– Some are mandatory, and some are optional.
– Some are clear and deterministic, others are vague and require reading and experimentation.
– Some make it easy to change vendors without breaking a sweat, while some will lock you into a specific implementation.

In this talk, I’ll explore the tradeoffs that a tester can make, and also explain the vendor point of view of the wild wild west of Selenium capabilities.

Many of you are doing THIS or suppose that doing. But at the testing conferences and meet-ups THIS is not acceptable. If this topic rises up, it is discussed quietly and almost in a whisper. There are people who decided to be in the opposition of THIS. But I was doing THIS openly for the last year and finally decided to share my experience.

This talk is focused on the method of White box testing. We will try to figure out what it is, how it works and why now it’s the best time to discard your prejudices and start using it.

I will talk about those parts of code where the most part of the bugs hidden, and why this method is one of the best ways of microservice testing.

Do you remember why your tests are written in language X? Because the project is being developed in language X? It was teamlead’s decision? Do you know good libraries and tools for language X? Maybe it’s time to try to write tests in language Y?

Yes, the tests’ language may differ from the project’s language. And if you are using Java, there are reasons why your tests’ language can be Kotlin. Don’t be afraid, Kotlin is not a rocket science. I’ll tell you how the features of this language make it possible to spend less time on the tests’ support. How strong typing can be similar to the dynamic one. And why your favorite libraries will stay with you. And, of course, the main thing – how to start. Where possible, a comparison with Java and Groovy will be made.

If you do test automation or you develop in Java/Groovy – come to my talk.

Why typed elements (like Buttons, Images, Checkboxes, Links etc.) are bad practice in UI tests? Or are non-typed elements (like SelenideElement for ´em all!) more evil? Is it better to have predefined Element hierarchy or not?

Alexei and Roman have delivered a sound theoretical battle earlier at Heisenbug-2016 Conference in Moscow. But were everything said just words? The both QA wrestlers are still standing behind their concepts and are willing to prove their point at live coding sessions before the judging audience! In this QA Battle the parties will solve problems using a typed framework JDI and an non-typed one – Selenide, commenting their actions. The audience will decide with their votes which of the concept was better.

As Selenium WebDriver is known to be a tool for browser automation and not for testing, we are doomed to use some testing frameworks to get efficient at automating web tests. But should every project reinvent the wheel? My answer is – “no!”.

Selenide (http://selenide.org) is a well-known matured (5 years old) web test automation framework with a large community, which has already solved almost every typical web automation problem. Just write your web tests in concise, easy to read manner, and concentrate on your business problems and not on taming your browsers.

In this workshop you will learn how to configure and start using Selenide from scratch in your project. While learning basics, we will be solving typical web automation tasks with real application exactly the way you would be doing it on your job!

Attendee should have basic understanding of Java, and how to write automated web test in general.

Software Developer in Test is relatively a new role in IT. Industry got more used to the titles like QA, Test Automation Engineer, Developer… Still this kind of roles distribution shows poor effectiveness and brings such things like huge suites of end to end tests as a quality gate. If you want to implement test pyramid, you should consider of such a role in your team.

What skills should have this person? What challenges and daily problems does he/she face? Can he/she change production code? This talk is about all these and many others connected questions. Perhaps, after this talk you will reconsider daily quality assuring approach.

This is all about making the testing process and outputs visible to the project team and to the company as a whole. We’ll try to answer questions below to make sense of our outputs and give meaningful metrics to our stakeholders:

– Why do you need to display test metrics in a project?
– What do you need to display?
– How do you display them?
– Some common mistakes to avoid on your information radars.

Design patters exist for years in software development. Some developers love them, some think they are useless. But design patters has very clear goals: describe common solutions for common problems, create shared language for community, improve understanding and reuse of existing approaches. Test automation has its own set of problems, so there is a set of helpful design patterns for this area. In this talk I will run through all known patterns and describe them in details with several practical samples.

In this master-class I will demonstrate how to start writing good functional tests using Java platform and WebDriver. We will cover JUnit basics, Java IDE usage, WebDriver API, good tests structure, approaches and techniques to make tests stable and reliable. All mentioned topics will be covered on real practical samples with live coding. Target audience is beginners and middle level, but some parts could look interesting even for experienced test automation engineers.

How to stop writing code and start writing automated tests. Based on real story.

(Example of how to create simple, readable and powerful REST API test automation solution with Spock and Groovy)

This talk is about how to test integrations in advertising project with multiple NoSql solutions, billions ads every day in real-time, many relations with external projects, doing it continuously and really quickly.

iOS Automation rapidly changed due to the evolution of Apple automation frameworks and Calabash community support. One year ago, we parallelised our iOS testing with multiple desktops – a large improvement in performance. Here is an account of the subsequent changes, including the removal of multiple desktops and the adoption of Apple’s new XCTest framework.

The automation community (Appium, Calabash etc.) faced a new challenge: how could we create iOS Automation Infrastructure with the new XCTest Framework?

I adopted Facebook solution (FBSimCtl and WebAgentDriver) into the multi-simulators approach. In this talk, I will discuss the full evolution path to Facebash Approach based on multi-simulators, Calabash server and Facebook Web Driver Agent.

If you are using Selenium as your test automation solution and you are testing web applications; there is a good chance that your company also has at least one mobile application. The common sense tells us that Selenium cannot be used for native apps for mobile, but with the times changing you’re probably asking yourself if you can at least use your Selenium skills with mobile test automation as well; the answer is a big “yes”.

Beyond any doubt, whether you establish a solid mobile test automation framework yourself or someone does it for you, it will bring many benefits. Some of the most critical ones include: a powerful foundation to cope with the mobile testing challenges and a systematic approach to shorten the time-to-market delivery by supporting the quality of your mobile products in preliminary stages.

My session mainly focuses on the challenges of mobile test automation (device variety, platform/OS diversity, operator/carrier issues, choice of native vs. HTML 5, emulator vs. real device testing and etc.) and it will guide attendees to practically utilise the solutions as Appium & Selendroid based on their experience on Selenium.

In this talk you will get ideas about how you can instrument test result information to provide actionable data, paving the way for more robust, reliable and timely test results.

By capturing this information over time, and when combined with visualization tools, we can answer different questions than with existing solutions (Allure/CI tool build history). Some examples of these are:

– Which tests are consistently flaky.
– What are the common causes of failure across tests.
– Which tests consistently take a long time to run.
– Using this information we can move away from the ‘re-run’ culture and better support continuous integration goals of having quick, reliable, deterministic tests.

This talk is about ways how use SoapUI tool for:

– Test Design.
– System testing and Integration testing.
– Test cases management.
– Continuous Integration.

All practices will be described with focus on REST services testing.

Responsive web design has become the preferred approach for building sites and apps that provide an optimal viewing and interaction experience on any phone, tablet, desktop or wearable device. However, automatically testing these responsive sites and apps can be quite a challenge, due to the need to cover all supported layouts, their respective navigation, and visible content.

In this session we will implement a complete Selenium-based automated test for a popular responsive website from scratch. You will learn how to effectively design responsive page objects, implement generic tests that work for all the layouts of your app, control the browser’s viewport size in order to accurately target layout transition points, incorporate layout-specific assertions in your tests, and visually validate the correctness of your app’s layout​. We will also share tips and best practices for test planning and execution.

White is a framework for automating rich client applications based on Win32, WinForms, WPF, Silverlight. It is a free, open-source framework, which can provide competition for commercial test automation tools for Windows. Also, it is great to use this framework with WebDriver for creation integration test on huge software solution which contain web and desktop parts.

In this talk we will delve into the biggest challenge that Test Automation developers face, finding elements robustness i.e. finding elements while the Application Under Test keeps changing. We’ll categorise methods, and show where developer fall, where machines fall (Record/Playback), and suggest a new way for locating elements, and analyze the skill-set required to overcome those difficulties.

In this talk I’ll share our successful experience of building reliable and scalable cross-platform automated acceptance test suite for a product called Citrix Concierge.

The product is built on a top of micro-services architecture. So, we will start from a brief introduction to a test strategy that was defined for this product. Then we’ll dig a bit more into acceptance testing and cover such topics as architecture and implementation of cross-platform acceptance test framework. Also, we I’ll show how we’re doing implementation of isolated tests, parallel execution, running at a scale using docker, etc.

We’ll be talking about technologies like: ruby, cucumber, rspec, capybara, appium, selenium webdriver, factory_girl, browsermob-proxy, faye.

Everyone knows about that one flaky test, or that one slow test, but no one ever fixes them. I will talk about how gathering metadata about your test runs can teach you about your tests, and help you test faster, smarter, and better. I will give examples from several years on the Developer Infrastructure team at Google, supporting the running of tens of millions of test cases using Selenium (and many more without) every day.

The Selenium Grid is a vital piece of infrastructure for any serious Selenium WebDriver test automation solution. Many organizations find the setup and maintenance of a Grid a difficult and expensive task. While outsourced solutions exist, they are too slow for any real professional Continuous Integration or Continuous Delivery workflow, and are expensive.

The Selenium Docker project provides a very easy to setup, deploy, maintain, and use solution for running a Selenium Grid on Docker supporting Firefox and Chrome on Linux. This talk will present how to use the Selenium Docker project, how to implement it as a PaaS solution, how to connect to Selenium Docker VMs and watch a test running for debugging, and extensions such as video recording.

This talk will cover the key pain points of UI automation: Speed, Reliability and Monitoring!

From speed perspective it will be about the means to achieve distributed execution of UI automation using AWS and Selenium Grid aka Autoscaled Distributed Automation with the goal of “Running all the test within the time taken by the slowest test case”. This is something we implemented and currently running all our 9000+ UI tests round the clock as part of our pipeline.

I will also cover key learnings on how we achieved reliability with UI automation, how and why we are focusing on reducing the overall UI Test Cases. Moving away from having 100+ jenkins job to only very few, storing UI results into MongoDB and NodeJS based reporting for easy monitoring of results!

Your web application use Web Players for showing Video, 3D objects, Flash,Flex or Unity 3D applications in browser? You can not configure Web Driver for access to UI Controls of your online Flash or Unity 3D game? Well, in this cases the best idea is to use Computer Vision technologies, and to connect Computer Vision framework with Web Driver.

OpenCVSharp is a free .NET library which give you opportunity to detect UI elements with help of image recognition algorithm. Using OpenCVSharp and Webdriver allows you to create test automation framework which can work with UI HTML elements of your web page and UI elements of your application which loaded in Web Player – Flash, Flex, Unity 3D – etc. Also, the above solution can be used for automation testing of web applications which connected with 3D graphics, Video, GIS, Games etc.

Used technologies: C#, OpenCV, WebDirver, WinApi32.

When you’re facing with an open API testing, you know that the testing itself is not a single problem. The main goal of this report is to share an approach of autoscalable testing with a bonus in automatic documentation generation. Java 8, Junit, Allure report are used as a tool set.

Everybody knows about test pyramid. It is always used on slides about test automation and presented on different conferences. But understanding test pyramid and its layers is not enough. There are more important questions: “how to find right balance for particular project?”, “how to build this pyramid from scratch for already running project?”, “what to do if you are limited in time and resources?”, “how to measure if your pyramid is good enough?”. In this talk I will bring some more “secret ingridients” on the table to make test pyramid concept really useful.

Fortunately today we have really easy and effective tool for Web UI Automation – Selenide for Java. In other languages the sky is not so blue and clean. 🙂 But world is changing and today we are meeting Selene – its alternative in Python. Selene also gathered ideas from other popular selenium framework in Java – htmlelements, which gives ability to construct your own custom elements in addition to simple WebElements. In this talk we will define the key features that make an abstract selenium wrapper really effective and will see how to use them with Selene in Python.

Test Oracles address a mutex problem that is faced by all teams who *successfully* practice continuous deployment. The problem test oracles address is: once your tests have run, and you have the results, how do you know what that means? In other words, what are you testing? What do your tests tell you? How do you know what you know?

Command line tools provide powerful data analytics when applied to a mature CI system and deployment pipeline. Such systems are known to be opaque — but the GNU/bash command line environment is powerful enough to solve those problems (with a little help from HTML5!).

A healthy CI system and deployment pipeline will naturally accrue a wealth of filesystem artifacts which can be mined for Test Oracles using command-line tools. Topics covered in depth will include:

• JSON testing and transformation with jq.
• Jenkins monitoring tips.
• Network analysis tools for Selenium developers.
• Visualizing xUnit test runner output.
• Analyzing your captured screenshots.
• Tips on introducing new dashboards to your team.

By providing yourself and your team not just with data but with actionable oracles, you will discover new reservoirs of activity that can safely become automated-rather-than-manual work.

There are a lot of books and publications about the continuous integration in the world. But in my experience it’s difficult to find information about how to open quality gates between automated tests and to continuous integration practice to in your current project. After reading several articles and even a couple of books you will understand how to work with it. But what next? I will share with you practical tips and tricks on how to lift iron curtain to your automated tests before a continuous quality practice today. It is for this reason why I am pleased to share with you my acquired experience in my presentation.

Automated visual testing is a major emerging trend in the dev/test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development/deployment lifecycle.

If you don’t know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

Functional tests are usually the slowest layer of automated tests for almost every product. They use product via UI, store data in real DB, integrate with external services and do other “slow” things. The first easy answer how to make them fast is to run in parallel. But in reality tests depends on the same data and intersect by some common functionality. In this talk we will review useful techniques and approaches how to win this battle.

In this talk we are going to highlight the architectural approach for test-driven automation we’ve successfully used in a number of legacy enterprise projects.
At first, when we got a legacy project to automate and a big team of testers, we’ve used the traditional way – automated on test-by-test basis. The results were lower of our expectations – we’ve missed the deadlines and we’ve encountered a number of problems on the way:

• different testers covered similar functionality in a different way, so we’ve got a lot of duplicated code;
• a method reused by different people sometimes turned out to be huge;
• the quality of manual test cases used as input was low, customer team feedback usually was delayed or missing, which slowed down automation in particular cases.

After first failures, we’ve came up with an approach that allowed us to avoid these problems in all following automation projects.

“Time is at once the most valuable and the most perishable of all our possessions”. Correspondingly we must know how to improve a quality of the project in the limitted timeframes. The goal of my presentation is improving an execution time of automated functional tests based on Selenium Webdriver, by using, for instance, parallel execution, scaling by distributing tests on several machines, creating strategy for generation of big sets of test data for typical project. I am pleased to share with you my acquired experience in this field.

BDD seems to be a nice idea. But I have never seen it really working.
The original idea behind BDD was collaboration with customer, and it simply does not work. What we get instead:

• Frameworks that make tests much more complexer.
• Tests structures that make tests much more harder to write.
• Organization structures that make learning much more difficult.
• Beautiful reports with screenshots that nobody reads.

Let’s talk about real cost of BDD and consider if it’s worth it.