TESTING

Copyright (c) 2011 The Chromium OS Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.

Intro
-----
We test shill using two sets of tests: unit tests, and integration
tests. The unit tests are built using Google Test [1] and Google Mock
[2]; the integration tests use autotest [3].

Running unit tests
------------------
- build the shill_unittest target
- run the resulting shill_unittest binary
- if you're using our ebuild scripts, you can do both by running
  (chroot)$ FEATURES="test" emerge-x86-generic shill
- run the unit tests from your host machine under gdb
  (chroot)$ FEATURES="test noclean" emerge-x86-generic shill
  (chroot)$ gdb_x86_local --board x86-generic \
                /build/x86-generic/tmp/portage/chromeos-base/shill-9999/work/shill-9999/shill_unittest
  (Of course if the unit tests segfaulted, you wouldn't need the emerge
  step since the build directory would have been retained in the course
  of the test failing.)

Running integration tests
-------------------------
- build a test image, suitable for a VM:
  (chroot) src/scripts$ ./build_packages --board=x86-generic
  (chroot) src/scripts$ ./build_image --board=x86-generic --withdev \
                            --test --noenable_rootfs_verification
  (chroot) src/scripts$ ./image_to_vm.sh --board=x86-generic --test_image

- start the VM
  (host)$ sudo kvm -m 1024 -vga std -pidfile /tmp/kvm.pid \
              -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \
              -hda <path to chroot>/src/build/images/x86-generic/latest/chromiumos_qemu_image.bin

- open a root shell on the VM console
  - in VM console, log in as guest
  - hit C-A-t to open a shell (use right control and alt, to avoid escaping
    out of VM's mouse capture)

  crosh> shell
  chronos@localhost / $ sudo bash

- load shill onto VM image
  localhost / # FEATURES="nostrip" gmerge shill

- stop flimflam, and start shill
  localhost / # stop flimflam
  localhost / # shill --v=1000 \
            --device-black-list=managed0,managed1,mon.managed0,mon.managed1
  (the black list keep shill from configuring the devices used by the hostapd
  processes.)

- run the tests
  (chroot) src/scripts$ ./run_remote_tests.sh --board=x86-generic \
                            --remote=127.0.0.1 --ssh_port 9222 \
                            --args="config_file=wifi_vm_config" WiFiManager

  To run a specific test out of the test suite, use test_pat option to --args.

  # Example: To just run the 035CheckWEPKeySyntax test:
  (chroot) src/scripts$ ./run_remote_tests.sh --board=x86-generic \
                            --remote=127.0.0.1 --ssh_port 9222 \
                            --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax" WiFiManager

- configuration note: if you use a different port
  (e.g. hostfwd=tcp::9223-:22), you'll need to change:
    - the ssh_port argument to run_remote_tests
    - the port numbers in
      <chroot>/third_party/autotest/files/client/config/wifi_vm_config

- debugging test failures
  - "grep shill /var/log/messages" for log messages
  - "grep wpa_supplicant /var/log/messages" for supplicant log messages
  - "wpa_debug debug" to increase supplicant log level
  - try resetting the test infrastructure
    - rmmod mac80211_hwsim mac80211 cfg80211
    - restart wpasupplicant
    - rm /tmp/hostapd-test.control/*

- additional test suites: we have a number of other WiFi test suites
  (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf,
  WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too
  relevant to shill (just yet), but the rest probably are.

[1] http://code.google.com/p/googletest/
[2] http://code.google.com/p/googlemock/
[3] http://autotest.kernel.org/,
    http://www.chromium.org/chromium-os/testing/testing-faq