mukesh agrawal | 8a3188d | 2011-12-01 20:56:44 +0000 | [diff] [blame] | 1 | Copyright (c) 2012 The Chromium OS Authors. All rights reserved. |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 2 | Use of this source code is governed by a BSD-style license that can be |
| 3 | found in the LICENSE file. |
| 4 | |
| 5 | Intro |
| 6 | ----- |
| 7 | We test shill using two sets of tests: unit tests, and integration |
| 8 | tests. The unit tests are built using Google Test [1] and Google Mock |
| 9 | [2]; the integration tests use autotest [3]. |
| 10 | |
| 11 | Running unit tests |
| 12 | ------------------ |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 13 | Here ${BOARD} is a valid board name, like link or x86-generic. |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 14 | - build the shill_unittest target |
| 15 | - run the resulting shill_unittest binary |
Paul Stewart | 85a1bad | 2011-10-21 22:36:38 -0700 | [diff] [blame] | 16 | - run the unit tests from your host machine under gdb |
Paul Stewart | 3de88a0 | 2014-12-05 07:55:14 -0800 | [diff] [blame] | 17 | (chroot)$ FEATURES=test emerge-${BOARD} shill |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 18 | (chroot)$ gdb_x86_local --board ${BOARD} \ |
| 19 | /build/${BOARD}/var/cache/portage/chromeos-base/platform2/out/Default/shill_unittest |
Paul Stewart | 85a1bad | 2011-10-21 22:36:38 -0700 | [diff] [blame] | 20 | (Of course if the unit tests segfaulted, you wouldn't need the emerge |
| 21 | step since the build directory would have been retained in the course |
| 22 | of the test failing.) |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 23 | - The emerge workflow given above is incremental. It uses ninja to rebuild only |
| 24 | relevant objects in the shill target. |
| 25 | - You can restrict the test runs to only shill unittests by using |
Paul Stewart | 3de88a0 | 2014-12-05 07:55:14 -0800 | [diff] [blame] | 26 | (chroot)$ P2_TEST_FILTER="shill::*" FEATURES=test emerge-${BOARD} shill |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 27 | The filter can be made more specific to include googletest filters like |
| 28 | "shill::CellularTest.StartGSMRegister" |
mukesh agrawal | cbfb34e | 2013-04-17 19:33:25 -0700 | [diff] [blame] | 29 | - if you want to set a breakpoint in gdb, make sure to include the shill |
| 30 | namespace. e.g., run |
| 31 | (cros-gdb) b shill::EthernetService::GetStorageIdentifier |
| 32 | Breakpoint 2 at 0x5555563cc270: file ethernet_service.cc, line 63. |
| 33 | rather than |
| 34 | (cros-gdb) b EthernetService::GetStorageIdentifier |
| 35 | Function "EthernetService::GetStorageIdentifier" not defined. |
| 36 | Make breakpoint pending on future shared library load? (y or [n]) n |
| 37 | |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 38 | - alternate build command: |
| 39 | - Another way to build which uses the emerge command behind the scenes: |
Paul Stewart | 3de88a0 | 2014-12-05 07:55:14 -0800 | [diff] [blame] | 40 | (chroot)$ cros_workon_make --board ${BOARD} shill |
mukesh agrawal | cbfb34e | 2013-04-17 19:33:25 -0700 | [diff] [blame] | 41 | - to see the actual compiler commands that are run: |
| 42 | (chroot)$ CFLAGS="-print-cmdline" cros_workon_make --reconf \ |
Paul Stewart | 3de88a0 | 2014-12-05 07:55:14 -0800 | [diff] [blame] | 43 | --board=${BOARD} shill |
mukesh agrawal | cbfb34e | 2013-04-17 19:33:25 -0700 | [diff] [blame] | 44 | - to abort compilation on the first error |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 45 | (chroot)$ MAKEFLAGS="--stop" cros_workon_make --test --board=${BOARD} \ |
Paul Stewart | 3de88a0 | 2014-12-05 07:55:14 -0800 | [diff] [blame] | 46 | --reconf shill |
| 47 | |
| 48 | Running unit tests with the address sanitizer |
| 49 | --------------------------------------------- |
| 50 | USE="asan clang wimax" TEST_FILTER="shill::*" emerge-${BOARD} shill |
| 51 | |
| 52 | This also turns on "wimax" and its tests, since this is disabled on most |
| 53 | platforms. |
mukesh agrawal | 76a87fd | 2013-04-19 18:17:49 -0700 | [diff] [blame] | 54 | |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 55 | Running integration tests |
| 56 | ------------------------- |
| 57 | - build a test image, suitable for a VM: |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 58 | (chroot) src/scripts$ ./build_packages --board=${BOARD} |
| 59 | (chroot) src/scripts$ ./build_image --board=${BOARD} \ |
Wade Guthrie | 8624e43 | 2012-04-11 11:12:00 -0700 | [diff] [blame] | 60 | --noenable_rootfs_verification test |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 61 | (chroot) src/scripts$ ./image_to_vm.sh --board=${BOARD} --test_image |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 62 | |
| 63 | - start the VM |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 64 | (host)$ sudo kvm -m 2048 -vga std -pidfile /tmp/kvm.pid \ |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 65 | -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \ |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 66 | -hda <path to chroot>/src/build/images/${BOARD}/latest/chromiumos_qemu_image.bin |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 67 | |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 68 | - DO NOT log in on the console. |
| 69 | (doing so will load a user profile onto shill's profile stack; this |
| 70 | interferes with the test profile that we use in the autotests) |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 71 | |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 72 | - if you've modified the source after building the image, update shill on |
| 73 | the image, and then restart shill: |
Wade Guthrie | 8624e43 | 2012-04-11 11:12:00 -0700 | [diff] [blame] | 74 | (chroot) src/scripts$ ./start_devserver |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 75 | (chroot) src/scripts$ ssh-keygen -R '[127.0.0.1]:9222' |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 76 | (chroot) src/scripts$ emerge-${BOARD} platform2 |
| 77 | (chroot) src/scripts$ cros deploy 127.0.0.1:9222 platform2 |
| 78 | |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 79 | (chroot) src/scripts$ ssh -p 9222 root@127.0.0.1 |
mukesh agrawal | 5b6a730 | 2012-11-19 14:25:14 -0800 | [diff] [blame] | 80 | localhost / # restart shill |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 81 | |
| 82 | - run the tests |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 83 | (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager |
| 84 | --args="config_file=wifi_vm_config" |
| 85 | --ssh_options="-p 9222" |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 86 | |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 87 | To run a specific test out of the test suite, use test_pat option to --args. |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 88 | # Example: To just run the 035CheckWEPKeySyntax test: |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 89 | (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager |
| 90 | --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax" |
| 91 | --ssh_options="-p 9222" |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 92 | |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 93 | - configuration note: if you use a different port |
| 94 | (e.g. hostfwd=tcp::9223-:22), you'll need to change: |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 95 | - the ssh_port argument to test_that |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 96 | - the port numbers in |
| 97 | <chroot>/third_party/autotest/files/client/config/wifi_vm_config |
| 98 | |
| 99 | - debugging test failures |
| 100 | - "grep shill /var/log/messages" for log messages |
| 101 | - "grep wpa_supplicant /var/log/messages" for supplicant log messages |
| 102 | - "wpa_debug debug" to increase supplicant log level |
| 103 | - try resetting the test infrastructure |
| 104 | - rmmod mac80211_hwsim mac80211 cfg80211 |
| 105 | - restart wpasupplicant |
| 106 | - rm /tmp/hostapd-test.control/* |
mukesh agrawal | c2f39fc | 2011-11-07 22:07:23 +0000 | [diff] [blame] | 107 | - examine autotest log files |
| 108 | - check how far the test got before it failed |
| 109 | $ grep -a ': step ' <test output>/<suite name>/<suite name>.<test name>/debug/<suite name>.<test name>.INFO |
| 110 | e.g. |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 111 | (chroot) $ grep -a ': step ' /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO |
mukesh agrawal | c2f39fc | 2011-11-07 22:07:23 +0000 | [diff] [blame] | 112 | - read the log file |
Prathmesh Prabhu | e7236b0 | 2014-02-13 17:54:56 -0800 | [diff] [blame] | 113 | (chroot) $ LESSOPEN= less /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO |
mukesh agrawal | c2f39fc | 2011-11-07 22:07:23 +0000 | [diff] [blame] | 114 | |
| 115 | (LESSOPEN= prevents less from misinterpreting the logs as binary files, |
| 116 | and piping them through hexdump.) |
mukesh agrawal | 70d3ebe | 2011-10-13 14:45:54 -0700 | [diff] [blame] | 117 | |
| 118 | - additional test suites: we have a number of other WiFi test suites |
| 119 | (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf, |
| 120 | WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too |
| 121 | relevant to shill (just yet), but the rest probably are. |
| 122 | |
| 123 | [1] http://code.google.com/p/googletest/ |
| 124 | [2] http://code.google.com/p/googlemock/ |
| 125 | [3] http://autotest.kernel.org/, |
Gaurav Shah | d52caa5 | 2011-10-25 10:46:03 -0700 | [diff] [blame] | 126 | http://www.chromium.org/chromium-os/testing/testing-faq |