mukesh agrawal | 8a3188d | 2011-12-01 20:56:44 +0000 | [diff] [blame] | 1 | Copyright (c) 2012 The Chromium OS Authors. All rights reserved. |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 2 | Use of this source code is governed by a BSD-style license that can be |
| 3 | found in the LICENSE file. |
| 4 | |
| 5 | Intro |
| 6 | ----- |
| 7 | We test shill using two sets of tests: unit tests, and integration |
| 8 | tests. The unit tests are built using Google Test [1] and Google Mock |
| 9 | [2]; the integration tests use autotest [3]. |
| 10 | |
| 11 | Running unit tests |
| 12 | ------------------ |
| 13 | - build the shill_unittest target |
| 14 | - run the resulting shill_unittest binary |
| 15 | - if you're using our ebuild scripts, you can do both by running |
Paul Stewart | 739742a | 2012-07-13 19:25:19 -0700 | [diff] [blame] | 16 | (chroot)$ cros_run_unit_tests -p shill |
Paul Stewart | 85a1bad | 2011-10-21 22:36:38 -0700 | [diff] [blame] | 17 | - run the unit tests from your host machine under gdb |
| 18 | (chroot)$ FEATURES="test noclean" emerge-x86-generic shill |
| 19 | (chroot)$ gdb_x86_local --board x86-generic \ |
| 20 | /build/x86-generic/tmp/portage/chromeos-base/shill-9999/work/shill-9999/shill_unittest |
| 21 | (Of course if the unit tests segfaulted, you wouldn't need the emerge |
| 22 | step since the build directory would have been retained in the course |
| 23 | of the test failing.) |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 24 | |
Chris Masone | 5756a36 | 2011-10-24 10:34:10 -0700 | [diff] [blame] | 25 | - for an incremental workflow, you can setup like this: |
| 26 | (chroot)$ cros_workon --board x86-generic shill |
Paul Stewart | 739742a | 2012-07-13 19:25:19 -0700 | [diff] [blame] | 27 | - incrementally build like this (note: this may not work if your |
| 28 | host architecture does not match that of the target): |
Chris Masone | 5756a36 | 2011-10-24 10:34:10 -0700 | [diff] [blame] | 29 | (chroot)$ CXXFLAGS=-g cros_workon_make --board x86-generic --reconf --test shill |
| 30 | - and then debug tests like this: |
| 31 | (chroot)$ gdb_x86_local --board x86-generic /path/to/checkout/src/platform/shill/shill_unittest |
| 32 | |
mukesh agrawal | 1590839 | 2011-11-16 18:29:25 +0000 | [diff] [blame] | 33 | - you can specify arguments to the unit tests like this: |
| 34 | - full workflow: |
| 35 | (chroot)$ GTEST_ARGS="--v=1000 --gtest_filter=WiFiPropertyTest.*" \ |
| 36 | FEATURES="test" emerge-x86-generic shill |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 37 | - incremental workflow: |
mukesh agrawal | 1590839 | 2011-11-16 18:29:25 +0000 | [diff] [blame] | 38 | (chroot)$ GTEST_ARGS="--v=1000 --gtest_filter=WiFiPropertyTest.*" \ |
| 39 | CXXFLAGS=-g cros_workon_make --board x86-generic --reconf --test shill |
| 40 | |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 41 | Running integration tests |
| 42 | ------------------------- |
| 43 | - build a test image, suitable for a VM: |
| 44 | (chroot) src/scripts$ ./build_packages --board=x86-generic |
Wade Guthrie | 8624e43 | 2012-04-11 11:12:00 -0700 | [diff] [blame] | 45 | (chroot) src/scripts$ ./build_image --board=x86-generic \ |
| 46 | --noenable_rootfs_verification test |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 47 | (chroot) src/scripts$ ./image_to_vm.sh --board=x86-generic --test_image |
| 48 | |
| 49 | - start the VM |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 50 | (host)$ sudo kvm -m 2048 -vga std -pidfile /tmp/kvm.pid \ |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 51 | -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \ |
| 52 | -hda <path to chroot>/src/build/images/x86-generic/latest/chromiumos_qemu_image.bin |
| 53 | |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 54 | - DO NOT log in on the console. |
| 55 | (doing so will load a user profile onto shill's profile stack; this |
| 56 | interferes with the test profile that we use in the autotests) |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 57 | |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 58 | - if you've modified the source after building the image, update shill on |
| 59 | the image, and then restart shill: |
Wade Guthrie | 8624e43 | 2012-04-11 11:12:00 -0700 | [diff] [blame] | 60 | (chroot) src/scripts$ ./start_devserver |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 61 | (chroot) src/scripts$ ssh-keygen -R '[127.0.0.1]:9222' |
| 62 | (chroot) src/scripts$ ssh -p 9222 root@127.0.0.1 |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 63 | localhost / # FEATURES="nostrip" gmerge shill |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 64 | localhost / # restart shill |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 65 | |
| 66 | - run the tests |
| 67 | (chroot) src/scripts$ ./run_remote_tests.sh --board=x86-generic \ |
| 68 | --remote=127.0.0.1 --ssh_port 9222 \ |
| 69 | --args="config_file=wifi_vm_config" WiFiManager |
| 70 | |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 71 | To run a specific test out of the test suite, use test_pat option to --args. |
| 72 | |
| 73 | # Example: To just run the 035CheckWEPKeySyntax test: |
| 74 | (chroot) src/scripts$ ./run_remote_tests.sh --board=x86-generic \ |
| 75 | --remote=127.0.0.1 --ssh_port 9222 \ |
| 76 | --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax" WiFiManager |
| 77 | |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 78 | - configuration note: if you use a different port |
| 79 | (e.g. hostfwd=tcp::9223-:22), you'll need to change: |
| 80 | - the ssh_port argument to run_remote_tests |
| 81 | - the port numbers in |
| 82 | <chroot>/third_party/autotest/files/client/config/wifi_vm_config |
| 83 | |
| 84 | - debugging test failures |
| 85 | - "grep shill /var/log/messages" for log messages |
| 86 | - "grep wpa_supplicant /var/log/messages" for supplicant log messages |
| 87 | - "wpa_debug debug" to increase supplicant log level |
| 88 | - try resetting the test infrastructure |
| 89 | - rmmod mac80211_hwsim mac80211 cfg80211 |
| 90 | - restart wpasupplicant |
| 91 | - rm /tmp/hostapd-test.control/* |
mukesh agrawal | c2f39fc | 2011-11-07 22:07:23 +0000 | [diff] [blame] | 92 | - examine autotest log files |
| 93 | - check how far the test got before it failed |
| 94 | $ grep -a ': step ' <test output>/<suite name>/<suite name>.<test name>/debug/<suite name>.<test name>.INFO |
| 95 | e.g. |
| 96 | (chroot) $ grep -a ': step ' /tmp/run_remote_tests.abcd/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO |
| 97 | - read the log file |
| 98 | (chroot) $ LESSOPEN= less /tmp/run_remote_tests.abcd/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO |
| 99 | |
| 100 | (LESSOPEN= prevents less from misinterpreting the logs as binary files, |
| 101 | and piping them through hexdump.) |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 102 | |
| 103 | - additional test suites: we have a number of other WiFi test suites |
| 104 | (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf, |
| 105 | WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too |
| 106 | relevant to shill (just yet), but the rest probably are. |
| 107 | |
| 108 | [1] http://code.google.com/p/googletest/ |
| 109 | [2] http://code.google.com/p/googlemock/ |
| 110 | [3] http://autotest.kernel.org/, |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 111 | http://www.chromium.org/chromium-os/testing/testing-faq |