1 2 Copyright (C) 2012 The Android Open Source Project 3 4 Licensed under the Apache License, Version 2.0 (the "License"); 5 you may not use this file except in compliance with the License. 6 You may obtain a copy of the License at 7 8 http://www.apache.org/licenses/LICENSE-2.0 9 10 Unless required by applicable law or agreed to in writing, software 11 distributed under the License is distributed on an "AS IS" BASIS, 12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 See the License for the specific language governing permissions and 14 limitations under the License. 15 16 Intro 17 ----- 18 We test shill using two sets of tests: unit tests, and integration 19 tests. The unit tests are built using Google Test [1] and Google Mock 20 [2]; the integration tests use autotest [3]. 21 22 Running unit tests for Chrome OS 23 -------------------------------- 24 Here ${BOARD} is a valid board name, like link or x86-generic. 25 - build the shill_unittest target 26 - run the resulting shill_unittest binary 27 - run the unit tests from your host machine under gdb 28 (chroot)$ FEATURES=test emerge-${BOARD} shill 29 (chroot)$ gdb_x86_local --board ${BOARD} \ 30 /build/${BOARD}/var/cache/portage/chromeos-base/platform2/out/Default/shill_unittest 31 (Of course if the unit tests segfaulted, you wouldn't need the emerge 32 step since the build directory would have been retained in the course 33 of the test failing.) 34 - The emerge workflow given above is incremental. It uses ninja to rebuild only 35 relevant objects in the shill target. 36 - You can restrict the test runs to only shill unittests by using 37 (chroot)$ P2_TEST_FILTER="shill::*" FEATURES=test emerge-${BOARD} shill 38 The filter can be made more specific to include googletest filters like 39 "shill::CellularTest.StartGSMRegister" 40 - if you want to set a breakpoint in gdb, make sure to include the shill 41 namespace. e.g., run 42 (cros-gdb) b shill::EthernetService::GetStorageIdentifier 43 Breakpoint 2 at 0x5555563cc270: file ethernet_service.cc, line 63. 44 rather than 45 (cros-gdb) b EthernetService::GetStorageIdentifier 46 Function "EthernetService::GetStorageIdentifier" not defined. 47 Make breakpoint pending on future shared library load? (y or [n]) n 48 49 - alternate build command: 50 - Another way to build which uses the emerge command behind the scenes: 51 (chroot)$ cros_workon_make --board ${BOARD} shill 52 - to see the actual compiler commands that are run: 53 (chroot)$ CFLAGS="-print-cmdline" cros_workon_make --reconf \ 54 --board=${BOARD} shill 55 - to abort compilation on the first error 56 (chroot)$ MAKEFLAGS="--stop" cros_workon_make --test --board=${BOARD} \ 57 --reconf shill 58 59 Running unit tests for Chrome OS with the address sanitizer 60 ----------------------------------------------------------- 61 USE="asan clang wimax" TEST_FILTER="shill::*" emerge-${BOARD} shill 62 63 This also turns on "wimax" and its tests, since this is disabled on most 64 platforms. 65 66 Running unit tests for Android 67 ------------------------------ 68 Currently, only native unit tests are supported on Android, meaning that unit tests 69 can only be run on the target device. 70 71 The test binary is located at /data/nativetest/shill_test/shill_test. 72 By default, it does not have the execute permissions due to inherited 73 directory security policy. Consequently, before running the unit tests, 74 you will need to grant execute permissions to the test binary using 75 the following command: 76 chmod +x /data/nativetest/shill_test/shill_test 77 78 To run all unit tests: 79 /data/nativetest/shill_test/shill_test 80 81 To run specific unit tests using filter: 82 /data/nativetest/shill_test/shill_test --gtest_filter=<TestClassName>.<TestName> 83 84 For example: 85 /data/nativetest/shill_test/shill_test --gtest_filter=WiFiMainTest.OnNewWiphy 86 /data/nativetest/shill_test/shill_test --gtest_filter=WiFiMainTest.* 87 88 Running integration tests 89 ------------------------- 90 - build a test image, suitable for a VM: 91 (chroot) src/scripts$ ./build_packages --board=${BOARD} 92 (chroot) src/scripts$ ./build_image --board=${BOARD} \ 93 --noenable_rootfs_verification test 94 (chroot) src/scripts$ ./image_to_vm.sh --board=${BOARD} --test_image 95 96 - start the VM 97 (host)$ sudo kvm -m 2048 -vga std -pidfile /tmp/kvm.pid \ 98 -net nic,model=virtio -net user,hostfwd=tcp::9222-:22 \ 99 -hda <path to chroot>/src/build/images/${BOARD}/latest/chromiumos_qemu_image.bin 100 101 - DO NOT log in on the console. 102 (doing so will load a user profile onto shill's profile stack; this 103 interferes with the test profile that we use in the autotests) 104 105 - if you've modified the source after building the image, update shill on 106 the image, and then restart shill: 107 (chroot) src/scripts$ ./start_devserver 108 (chroot) src/scripts$ ssh-keygen -R '[127.0.0.1]:9222' 109 (chroot) src/scripts$ emerge-${BOARD} platform2 110 (chroot) src/scripts$ cros deploy 127.0.0.1:9222 platform2 111 112 (chroot) src/scripts$ ssh -p 9222 root (a] 127.0.0.1 113 localhost / # restart shill 114 115 - run the tests 116 (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager 117 --args="config_file=wifi_vm_config" 118 --ssh_options="-p 9222" 119 120 To run a specific test out of the test suite, use test_pat option to --args. 121 # Example: To just run the 035CheckWEPKeySyntax test: 122 (chroot) src/scripts$ test_that 127.0.0.1 WiFiManager 123 --args="config_file=wifi_vm_config test_pat=035CheckWEPKeySyntax" 124 --ssh_options="-p 9222" 125 126 - configuration note: if you use a different port 127 (e.g. hostfwd=tcp::9223-:22), you'll need to change: 128 - the ssh_port argument to test_that 129 - the port numbers in 130 <chroot>/third_party/autotest/files/client/config/wifi_vm_config 131 132 - debugging test failures 133 - "grep shill /var/log/messages" for log messages 134 - "grep wpa_supplicant /var/log/messages" for supplicant log messages 135 - "wpa_debug debug" to increase supplicant log level 136 - try resetting the test infrastructure 137 - rmmod mac80211_hwsim mac80211 cfg80211 138 - restart wpasupplicant 139 - rm /tmp/hostapd-test.control/* 140 - examine autotest log files 141 - check how far the test got before it failed 142 $ grep -a ': step ' <test output>/<suite name>/<suite name>.<test name>/debug/<suite name>.<test name>.INFO 143 e.g. 144 (chroot) $ grep -a ': step ' /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO 145 - read the log file 146 (chroot) $ LESSOPEN= less /tmp/test_that_latest/network_WiFiRoaming/network_WiFiRoaming.002Suspend/debug/network_WiFiRoaming.002Suspend.INFO 147 148 (LESSOPEN= prevents less from misinterpreting the logs as binary files, 149 and piping them through hexdump.) 150 151 - additional test suites: we have a number of other WiFi test suites 152 (in addition to WiFiManager). these are: WiFiMatFunc, WiFiPerf, 153 WiFiRoaming, WiFiSecMat. the WiFiPerf tests are probably not too 154 relevant to shill (just yet), but the rest probably are. 155 156 [1] http://code.google.com/p/googletest/ 157 [2] http://code.google.com/p/googlemock/ 158 [3] http://autotest.kernel.org/, 159 http://www.chromium.org/chromium-os/testing/testing-faq 160
