==> Building on minun ==> Checking for remote environment... ==> Syncing package to remote host... sending incremental file list created directory packages/python-sh ./ .SRCINFO 809 100% 0.00kB/s 0:00:00 809 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=7/9) .nvchecker.toml 69 100% 67.38kB/s 0:00:00 69 100% 67.38kB/s 0:00:00 (xfr#2, to-chk=6/9) LICENSE 646 100% 630.86kB/s 0:00:00 646 100% 630.86kB/s 0:00:00 (xfr#3, to-chk=5/9) PKGBUILD 1,374 100% 1.31MB/s 0:00:00 1,374 100% 1.31MB/s 0:00:00 (xfr#4, to-chk=4/9) REUSE.toml 375 100% 366.21kB/s 0:00:00 375 100% 366.21kB/s 0:00:00 (xfr#5, to-chk=3/9) python-sh-2.2.2-2.log 840 100% 820.31kB/s 0:00:00 840 100% 820.31kB/s 0:00:00 (xfr#6, to-chk=2/9) LICENSES/ LICENSES/0BSD.txt -> ../LICENSE sent 2,803 bytes received 185 bytes 5,976.00 bytes/sec total size is 3,475 speedup is 1.16 ==> Running pkgctl build --arch riscv64 on remote host... ==> WARNING: invalid architecture: riscv64 ==> Updating pacman database cache [?25l:: Synchronizing package databases... core downloading... extra downloading... multilib downloading... [?25h==> Building python-sh  -> repo: extra  -> arch: riscv64  -> worker: felix-0 ==> Building python-sh for [extra] (riscv64) ]3008;start=a298903a834543bf94b6d965f312e206;user=root;hostname=minun.felixc.at;machineid=0ffe3ef7ad56462790c2861f2a747f6b;bootid=32d9c4b537a24816afcaf7574fd06933;pid=1652182;pidfdid=18425987;comm=systemd-nspawn;container=arch-nspawn-1652182;type=container\]11;?\]2;🔵 Container arch-nspawn-1652182 on minun.felixc.at\:: Synchronizing package databases... core downloading... extra downloading... :: Starting full system upgrade... there is nothing to do [!p]104[?7h]3008;end=a298903a834543bf94b6d965f312e206\==> Building in chroot for [extra] (riscv64)... ==> Synchronizing chroot copy [/var/lib/archbuild/extra-riscv64/root] -> [felix-0]...done ==> Making package: python-sh 2.2.2-2 (Fri Jan 23 21:43:44 2026) ==> Retrieving sources...  -> Cloning sh git repo... Cloning into bare repository '/home/felix/packages/python-sh/sh'... remote: Enumerating objects: 6977, done. remote: Counting objects: 0% (1/1031) remote: Counting objects: 1% (11/1031) remote: Counting objects: 2% (21/1031) remote: Counting objects: 3% (31/1031) remote: Counting objects: 4% (42/1031) remote: Counting objects: 5% (52/1031) remote: Counting objects: 6% (62/1031) remote: Counting objects: 7% (73/1031) remote: Counting objects: 8% (83/1031) remote: Counting objects: 9% (93/1031) remote: Counting objects: 10% (104/1031) remote: Counting objects: 11% (114/1031) remote: Counting objects: 12% (124/1031) remote: Counting objects: 13% (135/1031) remote: Counting objects: 14% (145/1031) remote: Counting objects: 15% (155/1031) remote: Counting objects: 16% (165/1031) remote: Counting objects: 17% (176/1031) remote: Counting objects: 18% (186/1031) remote: Counting objects: 19% (196/1031) remote: Counting objects: 20% (207/1031) remote: Counting objects: 21% (217/1031) remote: Counting objects: 22% (227/1031) remote: Counting objects: 23% (238/1031) remote: Counting objects: 24% (248/1031) remote: Counting objects: 25% (258/1031) remote: Counting objects: 26% (269/1031) remote: Counting objects: 27% (279/1031) remote: Counting objects: 28% (289/1031) remote: Counting objects: 29% (299/1031) remote: Counting objects: 30% (310/1031) remote: Counting objects: 31% (320/1031) remote: Counting objects: 32% (330/1031) remote: Counting objects: 33% (341/1031) remote: Counting objects: 34% (351/1031) remote: Counting objects: 35% (361/1031) remote: Counting objects: 36% (372/1031) remote: Counting objects: 37% (382/1031) remote: Counting objects: 38% (392/1031) remote: Counting objects: 39% (403/1031) remote: Counting objects: 40% (413/1031) remote: Counting objects: 41% (423/1031) remote: Counting objects: 42% (434/1031) remote: Counting objects: 43% (444/1031) remote: Counting objects: 44% (454/1031) remote: Counting objects: 45% (464/1031) remote: Counting objects: 46% (475/1031) remote: Counting objects: 47% (485/1031) remote: Counting objects: 48% (495/1031) remote: Counting objects: 49% (506/1031) remote: Counting objects: 50% (516/1031) remote: Counting objects: 51% (526/1031) remote: Counting objects: 52% (537/1031) remote: Counting objects: 53% (547/1031) remote: Counting objects: 54% (557/1031) remote: Counting objects: 55% (568/1031) remote: Counting objects: 56% (578/1031) remote: Counting objects: 57% (588/1031) remote: Counting objects: 58% (598/1031) remote: Counting objects: 59% (609/1031) remote: Counting objects: 60% (619/1031) remote: Counting objects: 61% (629/1031) remote: Counting objects: 62% (640/1031) remote: Counting objects: 63% (650/1031) remote: Counting objects: 64% (660/1031) remote: Counting objects: 65% (671/1031) remote: Counting objects: 66% (681/1031) remote: Counting objects: 67% (691/1031) remote: Counting objects: 68% (702/1031) remote: Counting objects: 69% (712/1031) remote: Counting objects: 70% (722/1031) remote: Counting objects: 71% (733/1031) remote: Counting objects: 72% (743/1031) remote: Counting objects: 73% (753/1031) remote: Counting objects: 74% (763/1031) remote: Counting objects: 75% (774/1031) remote: Counting objects: 76% (784/1031) remote: Counting objects: 77% (794/1031) remote: Counting objects: 78% (805/1031) remote: Counting objects: 79% (815/1031) remote: Counting objects: 80% (825/1031) remote: Counting objects: 81% (836/1031) remote: Counting objects: 82% (846/1031) remote: Counting objects: 83% (856/1031) remote: Counting objects: 84% (867/1031) remote: Counting objects: 85% (877/1031) remote: Counting objects: 86% (887/1031) remote: Counting objects: 87% (897/1031) remote: Counting objects: 88% (908/1031) remote: Counting objects: 89% (918/1031) remote: Counting objects: 90% (928/1031) remote: Counting objects: 91% (939/1031) remote: Counting objects: 92% (949/1031) remote: Counting objects: 93% (959/1031) remote: Counting objects: 94% (970/1031) remote: Counting objects: 95% (980/1031) remote: Counting objects: 96% (990/1031) remote: Counting objects: 97% (1001/1031) remote: Counting objects: 98% (1011/1031) remote: Counting objects: 99% (1021/1031) remote: Counting objects: 100% (1031/1031) remote: Counting objects: 100% (1031/1031), done. remote: Compressing objects: 0% (1/109) remote: Compressing objects: 1% (2/109) remote: Compressing objects: 2% (3/109) remote: Compressing objects: 3% (4/109) remote: Compressing objects: 4% (5/109) remote: Compressing objects: 5% (6/109) remote: Compressing objects: 6% (7/109) remote: Compressing objects: 7% (8/109) remote: Compressing objects: 8% (9/109) remote: Compressing objects: 9% (10/109) remote: Compressing objects: 10% (11/109) remote: Compressing objects: 11% (12/109) remote: Compressing objects: 12% (14/109) remote: Compressing objects: 13% (15/109) remote: Compressing objects: 14% (16/109) remote: Compressing objects: 15% (17/109) remote: Compressing objects: 16% (18/109) remote: Compressing objects: 17% (19/109) remote: Compressing objects: 18% (20/109) remote: Compressing objects: 19% (21/109) remote: Compressing objects: 20% (22/109) remote: Compressing objects: 21% (23/109) remote: Compressing objects: 22% (24/109) remote: Compressing objects: 23% (26/109) remote: Compressing objects: 24% (27/109) remote: Compressing objects: 25% (28/109) remote: Compressing objects: 26% (29/109) remote: Compressing objects: 27% (30/109) remote: Compressing objects: 28% (31/109) remote: Compressing objects: 29% (32/109) remote: Compressing objects: 30% (33/109) remote: Compressing objects: 31% (34/109) remote: Compressing objects: 32% (35/109) remote: Compressing objects: 33% (36/109) remote: Compressing objects: 34% (38/109) remote: Compressing objects: 35% (39/109) remote: Compressing objects: 36% (40/109) remote: Compressing objects: 37% (41/109) remote: Compressing objects: 38% (42/109) remote: Compressing objects: 39% (43/109) remote: Compressing objects: 40% (44/109) remote: Compressing objects: 41% (45/109) remote: Compressing objects: 42% (46/109) remote: Compressing objects: 43% (47/109) remote: Compressing objects: 44% (48/109) remote: Compressing objects: 45% (50/109) remote: Compressing objects: 46% (51/109) remote: Compressing objects: 47% (52/109) remote: Compressing objects: 48% (53/109) remote: Compressing objects: 49% (54/109) remote: Compressing objects: 50% (55/109) remote: Compressing objects: 51% (56/109) remote: Compressing objects: 52% (57/109) remote: Compressing objects: 53% (58/109) remote: Compressing objects: 54% (59/109) remote: Compressing objects: 55% (60/109) remote: Compressing objects: 56% (62/109) remote: Compressing objects: 57% (63/109) remote: Compressing objects: 58% (64/109) remote: Compressing objects: 59% (65/109) remote: Compressing objects: 60% (66/109) remote: Compressing objects: 61% (67/109) remote: Compressing objects: 62% (68/109) remote: Compressing objects: 63% (69/109) remote: Compressing objects: 64% (70/109) remote: Compressing objects: 65% (71/109) remote: Compressing objects: 66% (72/109) remote: Compressing objects: 67% (74/109) remote: Compressing objects: 68% (75/109) remote: Compressing objects: 69% (76/109) remote: Compressing objects: 70% (77/109) remote: Compressing objects: 71% (78/109) remote: Compressing objects: 72% (79/109) remote: Compressing objects: 73% (80/109) remote: Compressing objects: 74% (81/109) remote: Compressing objects: 75% (82/109) remote: Compressing objects: 76% (83/109) remote: Compressing objects: 77% (84/109) remote: Compressing objects: 78% (86/109) remote: Compressing objects: 79% (87/109) remote: Compressing objects: 80% (88/109) remote: Compressing objects: 81% (89/109) remote: Compressing objects: 82% (90/109) remote: Compressing objects: 83% (91/109) remote: Compressing objects: 84% (92/109) remote: Compressing objects: 85% (93/109) remote: Compressing objects: 86% (94/109) remote: Compressing objects: 87% (95/109) remote: Compressing objects: 88% (96/109) remote: Compressing objects: 89% (98/109) remote: Compressing objects: 90% (99/109) remote: Compressing objects: 91% (100/109) remote: Compressing objects: 92% (101/109) remote: Compressing objects: 93% (102/109) remote: Compressing objects: 94% (103/109) remote: Compressing objects: 95% (104/109) remote: Compressing objects: 96% (105/109) remote: Compressing objects: 97% (106/109) remote: Compressing objects: 98% (107/109) remote: Compressing objects: 99% (108/109) remote: Compressing objects: 100% (109/109) remote: Compressing objects: 100% (109/109), done. Receiving objects: 0% (1/6977) Receiving objects: 1% (70/6977) Receiving objects: 2% (140/6977) Receiving objects: 3% (210/6977) Receiving objects: 4% (280/6977) Receiving objects: 5% (349/6977) Receiving objects: 6% (419/6977) Receiving objects: 7% (489/6977) Receiving objects: 8% (559/6977) Receiving objects: 9% (628/6977) Receiving objects: 10% (698/6977) Receiving objects: 11% (768/6977) Receiving objects: 12% (838/6977) Receiving objects: 13% (908/6977) Receiving objects: 14% (977/6977) Receiving objects: 15% (1047/6977) Receiving objects: 16% (1117/6977) Receiving objects: 17% (1187/6977) Receiving objects: 18% (1256/6977) Receiving objects: 19% (1326/6977) Receiving objects: 20% (1396/6977) Receiving objects: 21% (1466/6977) Receiving objects: 22% (1535/6977) Receiving objects: 23% (1605/6977) Receiving objects: 24% (1675/6977) Receiving objects: 25% (1745/6977) Receiving objects: 26% (1815/6977) Receiving objects: 27% (1884/6977) Receiving objects: 28% (1954/6977) Receiving objects: 29% (2024/6977) Receiving objects: 30% (2094/6977) Receiving objects: 31% (2163/6977) Receiving objects: 32% (2233/6977) Receiving objects: 33% (2303/6977) Receiving objects: 34% (2373/6977) Receiving objects: 35% (2442/6977) Receiving objects: 36% (2512/6977) Receiving objects: 37% (2582/6977) Receiving objects: 38% (2652/6977) Receiving objects: 39% (2722/6977) Receiving objects: 40% (2791/6977) Receiving objects: 41% (2861/6977) Receiving objects: 42% (2931/6977) Receiving objects: 43% (3001/6977) Receiving objects: 44% (3070/6977) Receiving objects: 45% (3140/6977) Receiving objects: 46% (3210/6977) Receiving objects: 47% (3280/6977) Receiving objects: 48% (3349/6977) Receiving objects: 49% (3419/6977) Receiving objects: 50% (3489/6977) Receiving objects: 51% (3559/6977) Receiving objects: 52% (3629/6977) Receiving objects: 53% (3698/6977) Receiving objects: 54% (3768/6977) Receiving objects: 55% (3838/6977) Receiving objects: 56% (3908/6977) Receiving objects: 57% (3977/6977) Receiving objects: 58% (4047/6977) Receiving objects: 59% (4117/6977) Receiving objects: 60% (4187/6977) Receiving objects: 61% (4256/6977) Receiving objects: 62% (4326/6977) Receiving objects: 63% (4396/6977) Receiving objects: 64% (4466/6977) Receiving objects: 65% (4536/6977) Receiving objects: 66% (4605/6977) Receiving objects: 67% (4675/6977) Receiving objects: 68% (4745/6977) Receiving objects: 69% (4815/6977) Receiving objects: 70% (4884/6977) Receiving objects: 71% (4954/6977) Receiving objects: 72% (5024/6977) Receiving objects: 73% (5094/6977) Receiving objects: 74% (5163/6977) Receiving objects: 75% (5233/6977) Receiving objects: 76% (5303/6977) Receiving objects: 77% (5373/6977) Receiving objects: 78% (5443/6977) Receiving objects: 79% (5512/6977) Receiving objects: 80% (5582/6977) Receiving objects: 81% (5652/6977) Receiving objects: 82% (5722/6977) Receiving objects: 83% (5791/6977) Receiving objects: 84% (5861/6977) Receiving objects: 85% (5931/6977) Receiving objects: 86% (6001/6977) Receiving objects: 87% (6070/6977) Receiving objects: 88% (6140/6977) Receiving objects: 89% (6210/6977) Receiving objects: 90% (6280/6977) Receiving objects: 91% (6350/6977) Receiving objects: 92% (6419/6977) Receiving objects: 93% (6489/6977) Receiving objects: 94% (6559/6977) Receiving objects: 95% (6629/6977) Receiving objects: 96% (6698/6977) remote: Total 6977 (delta 959), reused 923 (delta 922), pack-reused 5946 (from 3) Receiving objects: 97% (6768/6977) Receiving objects: 98% (6838/6977) Receiving objects: 99% (6908/6977) Receiving objects: 100% (6977/6977) Receiving objects: 100% (6977/6977), 9.36 MiB | 34.00 MiB/s, done. Resolving deltas: 0% (0/4254) Resolving deltas: 1% (45/4254) Resolving deltas: 2% (86/4254) Resolving deltas: 3% (128/4254) Resolving deltas: 4% (172/4254) Resolving deltas: 5% (214/4254) Resolving deltas: 6% (256/4254) Resolving deltas: 7% (298/4254) Resolving deltas: 8% (341/4254) Resolving deltas: 9% (388/4254) Resolving deltas: 10% (427/4254) Resolving deltas: 11% (468/4254) Resolving deltas: 12% (511/4254) Resolving deltas: 13% (554/4254) Resolving deltas: 14% (596/4254) Resolving deltas: 15% (639/4254) Resolving deltas: 16% (681/4254) Resolving deltas: 17% (724/4254) Resolving deltas: 18% (767/4254) Resolving deltas: 19% (811/4254) Resolving deltas: 20% (851/4254) Resolving deltas: 21% (894/4254) Resolving deltas: 22% (936/4254) Resolving deltas: 23% (979/4254) Resolving deltas: 24% (1022/4254) Resolving deltas: 25% (1064/4254) Resolving deltas: 26% (1109/4254) Resolving deltas: 27% (1149/4254) Resolving deltas: 28% (1192/4254) Resolving deltas: 29% (1237/4254) Resolving deltas: 30% (1277/4254) Resolving deltas: 31% (1319/4254) Resolving deltas: 32% (1362/4254) Resolving deltas: 33% (1404/4254) Resolving deltas: 34% (1447/4254) Resolving deltas: 35% (1489/4254) Resolving deltas: 36% (1532/4254) Resolving deltas: 37% (1574/4254) Resolving deltas: 38% (1617/4254) Resolving deltas: 39% (1660/4254) Resolving deltas: 40% (1702/4254) Resolving deltas: 41% (1747/4254) Resolving deltas: 42% (1787/4254) Resolving deltas: 43% (1830/4254) Resolving deltas: 44% (1872/4254) Resolving deltas: 45% (1917/4254) Resolving deltas: 46% (1957/4254) Resolving deltas: 47% (2000/4254) Resolving deltas: 48% (2042/4254) Resolving deltas: 49% (2085/4254) Resolving deltas: 50% (2135/4254) Resolving deltas: 51% (2170/4254) Resolving deltas: 52% (2213/4254) Resolving deltas: 53% (2258/4254) Resolving deltas: 54% (2305/4254) Resolving deltas: 55% (2341/4254) Resolving deltas: 56% (2383/4254) Resolving deltas: 57% (2425/4254) Resolving deltas: 58% (2470/4254) Resolving deltas: 59% (2510/4254) Resolving deltas: 60% (2553/4254) Resolving deltas: 61% (2596/4254) Resolving deltas: 62% (2639/4254) Resolving deltas: 63% (2681/4254) Resolving deltas: 64% (2723/4254) Resolving deltas: 65% (2766/4254) Resolving deltas: 66% (2808/4254) Resolving deltas: 67% (2851/4254) Resolving deltas: 68% (2894/4254) Resolving deltas: 69% (2936/4254) Resolving deltas: 70% (2980/4254) Resolving deltas: 71% (3021/4254) Resolving deltas: 72% (3065/4254) Resolving deltas: 73% (3106/4254) Resolving deltas: 74% (3148/4254) Resolving deltas: 75% (3191/4254) Resolving deltas: 76% (3234/4254) Resolving deltas: 77% (3282/4254) Resolving deltas: 78% (3319/4254) Resolving deltas: 79% (3361/4254) Resolving deltas: 80% (3404/4254) Resolving deltas: 81% (3446/4254) Resolving deltas: 82% (3489/4254) Resolving deltas: 83% (3531/4254) Resolving deltas: 84% (3574/4254) Resolving deltas: 85% (3616/4254) Resolving deltas: 86% (3659/4254) Resolving deltas: 87% (3701/4254) Resolving deltas: 88% (3750/4254) Resolving deltas: 89% (3788/4254) Resolving deltas: 90% (3831/4254) Resolving deltas: 91% (3872/4254) Resolving deltas: 92% (3914/4254) Resolving deltas: 93% (3957/4254) Resolving deltas: 94% (3999/4254) Resolving deltas: 95% (4042/4254) Resolving deltas: 96% (4084/4254) Resolving deltas: 97% (4127/4254) Resolving deltas: 98% (4169/4254) Resolving deltas: 99% (4212/4254) Resolving deltas: 100% (4254/4254) Resolving deltas: 100% (4254/4254), done. ==> Validating source files with sha512sums... sh ... Passed ==> Validating source files with b2sums... sh ... Passed ]3008;start=8c2b248a2ffd412594cb3f9b2e026ef0;user=root;hostname=minun.felixc.at;machineid=0ffe3ef7ad56462790c2861f2a747f6b;bootid=32d9c4b537a24816afcaf7574fd06933;pid=1653254;pidfdid=18427059;comm=systemd-nspawn;container=arch-nspawn-1653254;type=container\]11;?\]2;🔵 Container arch-nspawn-1653254 on minun.felixc.at\==> Making package: python-sh 2.2.2-2 (Fri Jan 23 20:43:57 2026) ==> Checking runtime dependencies... ==> Installing missing dependencies... resolving dependencies... looking for conflicting packages... Package (2) New Version Net Change core/mpdecimal 4.0.1-1 0.31 MiB core/python 3.14.2-2 132.78 MiB Total Installed Size: 133.09 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing mpdecimal... installing python... Optional dependencies for python python-setuptools: for building Python packages using tooling that is usually bundled with Python python-pip: for installing Python packages using tooling that is usually bundled with Python python-pipx: for installing Python software not packaged on Arch Linux sqlite: for a default database integration [installed] xz: for lzma [installed] tk: for tkinter :: Running post-transaction hooks... (1/1) Arming ConditionNeedsUpdate... ==> Checking buildtime dependencies... ==> Installing missing dependencies... resolving dependencies... looking for conflicting packages... Package (27) New Version Net Change Download Size extra/perl-error 0.17030-3 0.04 MiB extra/perl-mailtools 2.22-3 0.10 MiB extra/perl-timedate 2.33-9 0.08 MiB extra/python-autocommand 2.2.2-9 0.08 MiB extra/python-fastjsonschema 2.21.2-2 0.28 MiB 0.05 MiB extra/python-iniconfig 2.1.0-3.1 0.05 MiB extra/python-jaraco.collections 5.1.0-3 0.11 MiB extra/python-jaraco.context 6.0.1-3 0.04 MiB extra/python-jaraco.functools 4.1.0-3 0.07 MiB extra/python-jaraco.text 4.0.0-4 0.08 MiB extra/python-lark-parser 1.3.1-2 1.39 MiB 0.27 MiB extra/python-more-itertools 10.8.0-2 0.73 MiB extra/python-packaging 26.0-1 0.89 MiB extra/python-platformdirs 4.5.1-3 0.28 MiB extra/python-pluggy 1.6.0-3.1 0.23 MiB extra/python-pygments 2.19.2-3 15.30 MiB extra/python-pyproject-hooks 1.2.0-6 0.11 MiB extra/python-typing_extensions 4.15.0-3 0.52 MiB extra/zlib-ng 2.2.5-1 0.21 MiB extra/git 2.52.0-2 28.75 MiB extra/lsof 4.99.5-2 0.34 MiB extra/python-build 1.4.0-1 0.24 MiB extra/python-installer 0.7.0-14 0.20 MiB extra/python-poetry-core 2.2.1-2 1.68 MiB 0.27 MiB extra/python-pytest 1:8.4.2-3 4.69 MiB extra/python-setuptools 1:80.9.0-4 8.03 MiB extra/python-wheel 0.45.1-4 0.30 MiB Total Download Size: 0.58 MiB Total Installed Size: 64.82 MiB :: Proceed with installation? [Y/n] :: Retrieving packages... python-lark-parser-1.3.1-2-any downloading... python-poetry-core-2.2.1-2-any downloading... python-fastjsonschema-2.21.2-2-any downloading... checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing zlib-ng... installing git... Optional dependencies for git git-zsh-completion: upstream zsh completion tk: gitk and git gui openssh: ssh transport and crypto man: show help with `git command --help` perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-cgi: gitweb (web interface) support python: git svn & git p4 [installed] subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] less: the default pager for git installing python-more-itertools... installing python-jaraco.functools... installing python-jaraco.context... installing python-autocommand... installing python-jaraco.text... Optional dependencies for python-jaraco.text python-inflect: for show-newlines script installing python-jaraco.collections... installing python-packaging... installing python-platformdirs... installing python-wheel... Optional dependencies for python-wheel python-keyring: for wheel.signatures python-xdg: for wheel.signatures python-setuptools: for legacy bdist_wheel subcommand [pending] installing python-setuptools... installing python-pyproject-hooks... installing python-build... Optional dependencies for python-build python-pip: to use as the Python package installer (default) python-uv: to use as the Python package installer python-virtualenv: to use virtualenv for build isolation installing python-installer... installing python-fastjsonschema... installing python-typing_extensions... installing python-lark-parser... Optional dependencies for python-lark-parser python-atomicwrites: for atomic_cache python-regex: for regex support installing python-poetry-core... installing lsof... installing python-iniconfig... installing python-pluggy... installing python-pygments... installing python-pytest... :: Running post-transaction hooks... (1/4) Creating system user accounts... Creating group 'git' with GID 969. Creating user 'git' (git daemon user) with UID 969 and GID 969. (2/4) Reloading system manager configuration... Skipped: Current root is not booted. (3/4) Arming ConditionNeedsUpdate... (4/4) Checking for old perl modules... ==> Retrieving sources... ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources... -> Creating working copy of sh git repo... Cloning into 'sh'... done. Switched to a new branch 'makepkg' ==> Starting build()... * Getting build dependencies for wheel... * Building wheel... Successfully built sh-2.2.2-py3-none-any.whl ==> Starting check()... ============================= test session starts ============================== platform linux -- Python 3.14.2, pytest-8.4.2, pluggy-1.6.0 rootdir: /build/python-sh/src/sh configfile: pyproject.toml collected 183 items / 1 deselected / 182 selected tests/sh_test.py ...F.F.FFFFFFFF.FFFF.FFFFFFFFFFFsFF.F.FF..F..FFFFFFFFFF [ 30%] .FF.FFFFF.FFFFFFFFFF.FF.FFF.FFFFFFFFFFFF.FFF..F.FFFFF.FFF.FFF.FFFFFF.FFF [ 69%] FF.FFFFFFFFFF.FFFFF.F.FFFFFFFFF..FF.FFFF..FFF....FFFF.F [100%] =================================== FAILURES =================================== ____________________ FunctionalTests.test_custom_separator _____________________ self = def test_custom_separator(self): py = create_tmp_test( """ import sys print(sys.argv[1]) """ ) opt = {"long-option": "underscore"} correct = "--long-option=custom=underscore" > out = python(py.name, opt, _long_sep="=custom=").strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:988: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e5550> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpn1rmmewm', '--long-option=custom=underscore'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_stringio_output _____________________ self = def test_stringio_output(self): import sh py = create_tmp_test( """ import sys sys.stdout.write(sys.argv[1]) """ ) out = StringIO() > sh.python(py.name, "testing 123", _out=out) tests/sh_test.py:2269: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpftzqnffs', 'testing 123'], stdin = None stdout = <_io.StringIO object at 0x7fa3a982cca0>, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_err_redirection _____________________ self = def test_err_redirection(self): import tempfile py = create_tmp_test( """ import sys import os sys.stdout.write("stdout") sys.stderr.write("stderr") """ ) file_obj = tempfile.NamedTemporaryFile() > p = python("-u", py.name, _err=file_obj) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1240: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3ac2786e0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpp1abbfw_'], stdin = None, stdout = None stderr = <_TemporaryFileWrapper file=<_io.BufferedRandom name='/tmp/tmp9_5d4bu4'>> call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_with_context_nested ___________________ self = def test_with_context_nested(self): echo_path = sh.echo._path with sh.echo.bake("test1", _with=True): with sh.echo.bake("test2", _with=True): > out = sh.echo("test3") ^^^^^^^^^^^^^^^^ tests/sh_test.py:1122: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7770> command = , parent_log = cmd = ['/usr/bin/echo', 'test1', '/usr/bin/echo', 'test2', '/usr/bin/echo', 'test3'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_no_pipe _________________________ self = def test_no_pipe(self): from sh import ls # calling a command regular should fill up the pipe_queue > p = ls(_return_cmd=True) ^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2520: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/ls'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_huge_piped_data _____________________ self = def test_huge_piped_data(self): from sh import tr stdin = tempfile.NamedTemporaryFile() data = "herpderp" * 4000 + "\n" stdin.write(data.encode()) stdin.flush() stdin.seek(0) > out = tr("[:upper:]", "[:lower:]", _in=tr("[:lower:]", "[:upper:]", _in=data)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2193: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/tr', '[:lower:]', '[:upper:]'] stdin = 'herpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpd...derpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderpherpderp\n' stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_stdout_callback_exit ___________________ self = def test_stdout_callback_exit(self): py = create_tmp_test( """ import sys import os for i in range(5): print(i) """ ) stdout = [] def agg(line): line = line.strip() stdout.append(line) if line == "2": return True > p = python("-u", py.name, _out=agg, _tee=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1573: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6cf0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmph79kjc6z'], stdin = None stdout = .agg at 0x7fa3a9f16770> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_bad_sig_raise_exception _________________ self = def test_bad_sig_raise_exception(self): # test all bad signal are correctly raised py = create_tmp_test( """ import time import sys time.sleep(2) sys.exit(1) """ ) for sig in SIGNALS_THAT_SHOULD_THROW_EXCEPTION: if sig == signal.SIGPIPE: continue sig_exception_name = f"SignalException_{sig}" sig_exception = getattr(sh, sig_exception_name) try: > p = python_bg(py.name) ^^^^^^^^^^^^^^^^^^ tests/sh_test.py:3172: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpcxd_o98q'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_no_close_fds _______________________ self = def test_no_close_fds(self): # guarantee some extra fds in our parent process that don't close on exec. we # have to explicitly do this because at some point (I believe python 3.4), # python started being more stringent with closing fds to prevent security # vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on # tempfile.TemporaryFile()s # # https://www.python.org/dev/peps/pep-0446/ tmp = [tempfile.TemporaryFile() for i in range(10)] for t in tmp: flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) flags &= ~fcntl.FD_CLOEXEC fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) py = create_tmp_test( """ import os print(len(os.listdir("/dev/fd"))) """ ) > out = python(py.name, _close_fds=False).strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:628: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpege8slqz'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_timeout_wait_overstep __________________ self = def test_timeout_wait_overstep(self): > p = sh.sleep(1, _bg=True) ^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2399: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/sleep', '1'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_none_arg _________________________ self = def test_none_arg(self): py = create_tmp_test( """ import sys print(sys.argv[1:]) """ ) maybe_arg = "some" > out = python(py.name, maybe_arg).strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:400: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpy9m43m9_', 'some'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_close_fds ________________________ self = def test_close_fds(self): # guarantee some extra fds in our parent process that don't close on exec. # we have to explicitly do this because at some point (I believe python 3.4), # python started being more stringent with closing fds to prevent security # vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on # tempfile.TemporaryFile()s # # https://www.python.org/dev/peps/pep-0446/ tmp = [tempfile.TemporaryFile() for i in range(10)] for t in tmp: flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) flags &= ~fcntl.FD_CLOEXEC fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) py = create_tmp_test( """ import os print(os.listdir("/dev/fd")) """ ) > out = python(py.name).strip() ^^^^^^^^^^^^^^^ tests/sh_test.py:656: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpqg_3ui2b'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_new_session_new_group __________________ self = def test_new_session_new_group(self): from threading import Event py = create_tmp_test( """ import os import time pid = os.getpid() pgid = os.getpgid(pid) sid = os.getsid(pid) stuff = [pid, pgid, sid] print(",".join([str(el) for el in stuff])) time.sleep(0.5) """ ) event = Event() def handle(run_asserts, line, stdin, p): pid, pgid, sid = line.strip().split(",") pid = int(pid) pgid = int(pgid) sid = int(sid) test_pid = os.getpgid(os.getpid()) self.assertEqual(p.pid, pid) self.assertEqual(p.pgid, pgid) self.assertEqual(pgid, p.get_pgid()) self.assertEqual(p.sid, sid) self.assertEqual(sid, p.get_sid()) run_asserts(pid, pgid, sid, test_pid) event.set() def session_true_group_false(pid, pgid, sid, test_pid): self.assertEqual(pid, sid) self.assertEqual(pid, pgid) > p = python( py.name, _out=partial(handle, session_true_group_false), _new_session=True ) tests/sh_test.py:2800: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpop6ts61_'], stdin = None stdout = functools.partial(.handle at 0x7fa3a9448d50>, .session_true_group_false at 0x7fa3a9448e00>) stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_no_err __________________________ self = def test_no_err(self): py = create_tmp_test( """ import sys sys.stdout.write("stdout") sys.stderr.write("stderr") """ ) > p = python(py.name, _no_err=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2498: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e74d0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpl8jw5a1b'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_incremental_composition _________________ self = def test_incremental_composition(self): py1 = create_tmp_test( """ import sys print(int(sys.argv[1]) * 2) """ ) py2 = create_tmp_test( """ import sys print(int(sys.stdin.read()) + 1) """ ) > res = python(py2.name, _in=python(py1.name, 8, _piped=True)).strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:931: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b4d70> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpiclkvzhd'] stdin = stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_non_ascii_error _____________________ self = def test_non_ascii_error(self): from sh import ErrorReturnCode, ls test = "/á" > self.assertRaises(ErrorReturnCode, ls, test, _encoding="utf8") tests/sh_test.py:2451: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_ok_code _________________________ self = def test_ok_code(self): from sh import ErrorReturnCode_1, ErrorReturnCode_2, ls exc_to_test = ErrorReturnCode_2 code_to_pass = 2 if IS_MACOS: exc_to_test = ErrorReturnCode_1 code_to_pass = 1 > self.assertRaises(exc_to_test, ls, "/aofwje/garogjao4a/eoan3on") tests/sh_test.py:376: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_grandchild_no_sighup ___________________ self = def test_grandchild_no_sighup(self): import time # child process that will write to a file if it receives a SIGHUP child = create_tmp_test( """ import signal import sys import time output_file = sys.argv[1] with open(output_file, "w") as f: def handle_sighup(signum, frame): f.write("got signal %d" % signum) sys.exit(signum) signal.signal(signal.SIGHUP, handle_sighup) time.sleep(2) f.write("made it!\\n") """ ) # the parent that will terminate before the child writes to the output # file, potentially causing a SIGHUP parent = create_tmp_test( """ import os import time import sys child_file = sys.argv[1] output_file = sys.argv[2] python_name = os.path.basename(sys.executable) os.spawnlp(os.P_NOWAIT, python_name, python_name, child_file, output_file) time.sleep(1) # give child a chance to set up """ ) output_file = tempfile.NamedTemporaryFile(delete=True) > python(parent.name, child.name, output_file.name) tests/sh_test.py:3087: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aae178c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpuecrvjg3', '/tmp/tmpd5s3mofq', '/tmp/tmplgh2ensv'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_stdout_callback_no_wait _________________ self = def test_stdout_callback_no_wait(self): import time py = create_tmp_test( """ import sys import os import time for i in range(5): print(i) time.sleep(.5) """ ) stdout = [] def agg(line): stdout.append(line) > python("-u", py.name, _out=agg, _bg=True) tests/sh_test.py:1465: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3ac2786e0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp42073g2d'], stdin = None stdout = .agg at 0x7fa3a949bed0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_short_option _______________________ self = def test_short_option(self): from sh import sh > s1 = sh(c="echo test").strip() ^^^^^^^^^^^^^^^^^ tests/sh_test.py:937: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/sh', '-c', 'echo test'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_custom_separator_space __________________ self = def test_custom_separator_space(self): py = create_tmp_test( """ import sys print(str(sys.argv[1:])) """ ) opt = {"long-option": "space"} correct = ["--long-option", "space"] > out = python(py.name, opt, _long_sep=" ").strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1006: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3ac2786e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmplle6_xa4', '--long-option', 'space'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_multiple_pipes ______________________ self = def test_multiple_pipes(self): import time py = create_tmp_test( """ import sys import os import time for l in "andrew": sys.stdout.write(l) time.sleep(.2) """ ) inc_py = create_tmp_test( """ import sys while True: letter = sys.stdin.read(1) if not letter: break sys.stdout.write(chr(ord(letter)+1)) """ ) def inc(*args, **kwargs): return python("-u", inc_py.name, *args, **kwargs) class Derp: def __init__(self): self.times = [] self.stdout = [] self.last_received = None def agg(self, line): self.stdout.append(line.strip()) now = time.time() if self.last_received: self.times.append(now - self.last_received) self.last_received = now derp = Derp() > p = inc( _in=inc( _in=inc(_in=python("-u", py.name, _piped=True), _piped=True), _piped=True, ), _out=derp.agg, ) tests/sh_test.py:478: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:461: in inc return python("-u", inc_py.name, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp7h513k7f'] stdin = stdout = .Derp.agg of .Derp object at 0x7fa3aa7e57f0>> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_done_callback ______________________ self = def test_done_callback(self): import time class Callback: def __init__(self): self.called = False self.exit_code = None self.success = None def __call__(self, p, success, exit_code): self.called = True self.exit_code = exit_code self.success = success py = create_tmp_test( """ from time import time, sleep sleep(1) print(time()) """ ) callback = Callback() > p = python(py.name, _done=callback, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2713: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5a90> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpjhjk46h0'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_stdin_unbuffered_bufsize _________________ self = def test_stdin_unbuffered_bufsize(self): from time import sleep # this tries to receive some known data and measures the time it takes # to receive it. since we're flushing by newline, we should only be # able to receive the data when a newline is fed in py = create_tmp_test( """ import sys from time import time started = time() data = sys.stdin.read(len("testing")) waited = time() - started sys.stdout.write(data + "\\n") sys.stdout.write(str(waited) + "\\n") started = time() data = sys.stdin.read(len("done")) waited = time() - started sys.stdout.write(data + "\\n") sys.stdout.write(str(waited) + "\\n") sys.stdout.flush() """ ) def create_stdin(): yield "test" sleep(1) yield "ing" sleep(1) yield "done" > out = python(py.name, _in=create_stdin(), _in_bufsize=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2916: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5be0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp4qcn81uw'] stdin = .create_stdin at 0x7fa3a981e5a0> stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_ok_code_exception ____________________ self = def test_ok_code_exception(self): from sh import ErrorReturnCode_0 py = create_tmp_test("exit(0)") > self.assertRaises(ErrorReturnCode_0, python, py.name, _ok_code=2) tests/sh_test.py:390: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_async_exc ________________________ self = def test_async_exc(self): py = create_tmp_test("""exit(34)""") async def producer(): await python(py.name, _async=True, _return_cmd=False) > self.assertRaises(sh.ErrorReturnCode_34, asyncio.run, producer()) tests/sh_test.py:1752: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1750: in producer await python(py.name, _async=True, _return_cmd=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_async_iter ________________________ self = def test_async_iter(self): py = create_tmp_test( """ for i in range(5): print(i) """ ) # this list will prove that our coroutines are yielding to eachother as each # line is produced alternating = [] async def producer(q): async for line in python(py.name, _iter=True): alternating.append(1) await q.put(int(line.strip())) await q.put(None) async def consumer(q): while True: line = await q.get() if line is None: return alternating.append(2) async def main(): q = AQueue() await asyncio.gather(producer(q), consumer(q)) > asyncio.run(main()) tests/sh_test.py:1784: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1782: in main await asyncio.gather(producer(q), consumer(q)) tests/sh_test.py:1767: in producer async for line in python(py.name, _iter=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpxyhctgi8'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_doesnt_execute_directories ________________ self = def test_doesnt_execute_directories(self): save_path = os.environ["PATH"] bin_dir1 = tempfile.mkdtemp() bin_dir2 = tempfile.mkdtemp() gcc_dir1 = os.path.join(bin_dir1, "gcc") gcc_file2 = os.path.join(bin_dir2, "gcc") try: os.environ["PATH"] = os.pathsep.join((bin_dir1, bin_dir2)) # a folder named 'gcc', its executable, but should not be # discovered by internal which(1)-clone os.makedirs(gcc_dir1) # an executable named gcc -- only this should be executed bunk_header = "#!/bin/sh\necho $*" with open(gcc_file2, "w") as h: h.write(bunk_header) os.chmod(gcc_file2, int(0o755)) from sh import gcc self.assertEqual(gcc._path, gcc_file2) self.assertEqual( > gcc("no-error", _return_cmd=True).stdout.strip(), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ b"no-error", ) tests/sh_test.py:811: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/tmp/tmpk77gtr_8/gcc', 'no-error'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_command_wrapper_equivalence _______________ self = def test_command_wrapper_equivalence(self): from sh import Command, ls, which > self.assertEqual(Command(str(which("ls")).strip()), ls) ^^^^^^^^^^^ tests/sh_test.py:788: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/which', 'ls'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_iter_generator ______________________ self = def test_iter_generator(self): py = create_tmp_test( """ import sys import os import time for i in range(42): print(i) sys.stdout.flush() """ ) out = [] > for line in python(py.name, _iter=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1709: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7770> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpeem2ihrz'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_change_stdout_buffering _________________ self = def test_change_stdout_buffering(self): py = create_tmp_test( """ import sys import os # this proves that we won't get the output into our callback until we send # a newline sys.stdout.write("switch ") sys.stdout.flush() sys.stdout.write("buffering\\n") sys.stdout.flush() sys.stdin.read(1) sys.stdout.write("unbuffered") sys.stdout.flush() # this is to keep the output from being flushed by the process ending, which # would ruin our test. we want to make sure we get the string "unbuffered" # before the process ends, without writing a newline sys.stdin.read(1) """ ) d = { "newline_buffer_success": False, "unbuffered_success": False, } def interact(line, stdin, process): line = line.strip() if not line: return if line == "switch buffering": d["newline_buffer_success"] = True process.change_out_bufsize(0) stdin.put("a") elif line == "unbuffered": stdin.put("b") d["unbuffered_success"] = True return True # start with line buffered stdout > pw_stars = python("-u", py.name, _out=interact, _out_bufsize=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2340: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp2_45uw02'], stdin = None stdout = .interact at 0x7fa3a99b4930> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_custom_timeout_signal __________________ self = def test_custom_timeout_signal(self): import signal from sh import TimeoutException py = create_tmp_test( """ import time time.sleep(3) """ ) try: > python(py.name, _timeout=1, _timeout_signal=signal.SIGHUP) tests/sh_test.py:2983: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp1izje7dd'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________________ FunctionalTests.test_cwd ___________________________ self = def test_cwd(self): from os.path import realpath from sh import pwd > self.assertEqual(str(pwd(_cwd="/tmp")), realpath("/tmp") + "\n") ^^^^^^^^^^^^^^^^ tests/sh_test.py:2159: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/pwd'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_stdin_from_string ____________________ self = def test_stdin_from_string(self): from sh import sed self.assertEqual( > sed(_in="one test three", e="s/test/two/").strip(), "one two three" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) tests/sh_test.py:365: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/sed', '-e', 's/test/two/'], stdin = 'one test three' stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_no_arg __________________________ self = def test_no_arg(self): import pwd from sh import whoami > u1 = whoami().strip() ^^^^^^^^ tests/sh_test.py:696: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7770> command = , parent_log = cmd = ['/usr/bin/whoami'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_async_iter_exc ______________________ self = def test_async_iter_exc(self): py = create_tmp_test( """ for i in range(5): print(i) exit(34) """ ) lines = [] async def producer(): async for line in python(py.name, _async=True): lines.append(int(line.strip())) > self.assertRaises(sh.ErrorReturnCode_34, asyncio.run, producer()) tests/sh_test.py:1802: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1799: in producer async for line in python(py.name, _async=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_sigpipe _________________________ self = def test_sigpipe(self): py1 = create_tmp_test( """ import sys import os import time import signal # by default, python disables SIGPIPE, in favor of using IOError exceptions, so # let's put that back to the system default where we terminate with a signal # exit code signal.signal(signal.SIGPIPE, signal.SIG_DFL) for letter in "andrew": time.sleep(0.6) print(letter) """ ) py2 = create_tmp_test( """ import sys import os import time while True: line = sys.stdin.readline() if not line: break print(line.strip().upper()) exit(0) """ ) p1 = python("-u", py1.name, _piped="out") > p2 = python( "-u", py2.name, _in=p1, ) tests/sh_test.py:1970: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6cf0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmphgbbxxzw'] stdin = stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_done_cb_exc _______________________ self = def test_done_cb_exc(self): from sh import ErrorReturnCode class Callback: def __init__(self): self.called = False self.success = None def __call__(self, p, success, exit_code): self.success = success self.called = True py = create_tmp_test("exit(1)") callback = Callback() try: > p = python(py.name, _done=callback, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2851: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpfn_aqt9o'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_custom_long_prefix ____________________ self = def test_custom_long_prefix(self): py = create_tmp_test( """ import sys print(sys.argv[1]) """ ) > out = python( py.name, {"long-option": "underscore"}, _long_prefix="-custom-" ).strip() tests/sh_test.py:1017: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpf5jgilai', '-custom-long-option=underscore'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_timeout_wait _______________________ self = def test_timeout_wait(self): > p = sh.sleep(3, _bg=True) ^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2395: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/sleep', '3'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_stdout_callback_terminate ________________ self = def test_stdout_callback_terminate(self): import signal py = create_tmp_test( """ import sys import os import time for i in range(5): print(i) time.sleep(.5) """ ) stdout = [] def agg(line, stdin, process): line = line.strip() stdout.append(line) if line == "3": process.terminate() return True import sh caught_signal = False try: > p = python("-u", py.name, _out=agg, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1607: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpo866k253'], stdin = None stdout = .agg at 0x7fa3a9f14bf0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_callable_stdin ______________________ self = def test_callable_stdin(self): py = create_tmp_test( """ import sys sys.stdout.write(sys.stdin.read()) """ ) def create_stdin(): state = {"count": 0} def stdin(): count = state["count"] if count == 4: return None state["count"] += 1 return str(count) return stdin > out = pythons(py.name, _in=create_stdin()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2879: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmprnajn630'] stdin = .create_stdin..stdin at 0x7fa3a99b4f60> stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_stdout_callback _____________________ self = def test_stdout_callback(self): py = create_tmp_test( """ import sys import os for i in range(5): print(i) """ ) stdout = [] def agg(line): stdout.append(line) > p = python("-u", py.name, _out=agg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1440: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpgxw0de5w'], stdin = None stdout = .agg at 0x7fa3a93dec40> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_exit_code ________________________ self = def test_exit_code(self): from sh import ErrorReturnCode_3 py = create_tmp_test( """ exit(3) """ ) > self.assertRaises(ErrorReturnCode_3, python, py.name) tests/sh_test.py:311: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_timeout _________________________ self = def test_timeout(self): from time import time import sh sleep_for = 3 timeout = 1 started = time() try: > sh.sleep(sleep_for, _timeout=timeout).wait() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2380: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5fd0> command = , parent_log = cmd = ['/usr/bin/sleep', '3'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_generator_and_callback __________________ self = def test_generator_and_callback(self): py = create_tmp_test( """ import sys import os for i in range(42): sys.stderr.write(str(i * 2)+"\\n") print(i) """ ) stderr = [] def agg(line): stderr.append(int(line.strip())) out = [] > for line in python("-u", py.name, _iter=True, _err=agg): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2058: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5be0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp7bapxd03'], stdin = None, stdout = None stderr = .agg at 0x7fa3a99b40f0> call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_patched_glob _______________________ self = def test_patched_glob(self): from glob import glob py = create_tmp_test( """ import sys print(sys.argv[1:]) """ ) files = glob("*.faowjefoajweofj") > out = python(py.name, files).strip() ^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:323: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b4ec0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpnh25f5kx', '*.faowjefoajweofj'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_timeout_overstep _____________________ self = def test_timeout_overstep(self): started = time.time() > sh.sleep(1, _timeout=5) tests/sh_test.py:2390: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5d30> command = , parent_log = cmd = ['/usr/bin/sleep', '1'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________________ FunctionalTests.test_pushd __________________________ self = def test_pushd(self): """test basic pushd functionality""" child = realpath(tempfile.mkdtemp()) > old_wd1 = sh.pwd().strip() ^^^^^^^^ tests/sh_test.py:2645: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5a90> command = , parent_log = cmd = ['/usr/bin/pwd'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_quote_escaping ______________________ self = def test_quote_escaping(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() options, args = parser.parse_args() print(args) """ ) > out = python(py.name, "one two three").strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:416: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b57f0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpi9i90r0d', 'one two three'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_tty_input ________________________ self = def test_tty_input(self): py = create_tmp_test( """ import sys import os if os.isatty(sys.stdin.fileno()): sys.stdout.write("password?\\n") sys.stdout.flush() pw = sys.stdin.readline().strip() sys.stdout.write("%s\\n" % ("*" * len(pw))) sys.stdout.flush() else: sys.stdout.write("no tty attached!\\n") sys.stdout.flush() """ ) test_pw = "test123" expected_stars = "*" * len(test_pw) d = {} def password_enterer(line, stdin): line = line.strip() if not line: return if line == "password?": stdin.put(test_pw + "\n") elif line.startswith("*"): d["stars"] = line return True > pw_stars = python(py.name, _tty_in=True, _out=password_enterer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2230: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'pid'") raised in repr()] OProc object at 0x7fa3a94b6270> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpjhc47lcx'], stdin = None stdout = .password_enterer at 0x7fa3a99b71c0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: > self.ctty = os.ttyname(self._stdin_child_fd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E OSError: [Errno 25] Inappropriate ioctl for device sh.py:1963: OSError _____________ FunctionalTests.test_stdout_callback_line_unbuffered _____________ self = def test_stdout_callback_line_unbuffered(self): py = create_tmp_test( """ import sys import os for i in range(5): print("herpderp") """ ) stdout = [] def agg(char): stdout.append(char) > p = python("-u", py.name, _out=agg, _out_bufsize=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1508: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aae178c0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpzh5d_24u'], stdin = None stdout = .agg at 0x7fa3a93de1f0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_manual_stdin_iterable __________________ self = def test_manual_stdin_iterable(self): from sh import tr test = ["testing\n", "herp\n", "derp\n"] > out = tr("[:lower:]", "[:upper:]", _in=test) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:500: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e5550> command = , parent_log = cmd = ['/usr/bin/tr', '[:lower:]', '[:upper:]'] stdin = ['testing\n', 'herp\n', 'derp\n'], stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_ok_code_none _______________________ self = def test_ok_code_none(self): py = create_tmp_test("exit(0)") > python(py.name, _ok_code=None) tests/sh_test.py:384: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpshzdc3he'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_signal_group _______________________ self = def test_signal_group(self): child = create_tmp_test( """ import time time.sleep(3) """ ) parent = create_tmp_test( """ import sys import sh python = sh.Command(sys.executable) p = python("{child_file}", _bg=True, _new_session=False) print(p.pid) print(p.process.pgid) p.wait() """, child_file=child.name, ) def launch(): p = python(parent.name, _bg=True, _iter=True, _new_group=True) child_pid = int(next(p).strip()) child_pgid = int(next(p).strip()) parent_pid = p.pid parent_pgid = p.process.pgid return p, child_pid, child_pgid, parent_pid, parent_pgid def assert_alive(pid): os.kill(pid, 0) def assert_dead(pid): self.assert_oserror(errno.ESRCH, os.kill, pid, 0) # first let's prove that calling regular SIGKILL on the parent does # nothing to the child, since the child was launched in the same process # group (_new_session=False) and the parent is not a controlling process > p, child_pid, child_pgid, parent_pid, parent_pgid = launch() ^^^^^^^^ tests/sh_test.py:2616: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:2599: in launch p = python(parent.name, _bg=True, _iter=True, _new_group=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e4830> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp8depx1jj'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_timeout_wait_negative __________________ self = def test_timeout_wait_negative(self): > p = sh.sleep(3, _bg=True) ^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2403: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e46e0> command = , parent_log = cmd = ['/usr/bin/sleep', '3'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_binary_input _______________________ self = def test_binary_input(self): py = create_tmp_test( """ import sys data = sys.stdin.read() sys.stdout.write(data) """ ) data = b"1234" > out = pythons(py.name, _in=data) ^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1134: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpfe5k3odq'], stdin = b'1234', stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_false_bool_ignore ____________________ self = def test_false_bool_ignore(self): py = create_tmp_test( """ import sys print(sys.argv[1:]) """ ) test = True > self.assertEqual(python(py.name, test and "-n").strip(), "['-n']") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:894: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpiqflkr1u', '-n'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_async_return_cmd _____________________ self = def test_async_return_cmd(self): py = create_tmp_test( """ import sys sys.exit(0) """ ) async def main(): result = await python(py.name, _async=True, _return_cmd=True) self.assertIsInstance(result, sh.RunningCommand) result_str = await python(py.name, _async=True, _return_cmd=False) self.assertIsInstance(result_str, str) > asyncio.run(main()) tests/sh_test.py:1818: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1813: in main result = await python(py.name, _async=True, _return_cmd=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp1pt0eig2'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': True, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_async_return_cmd_exc ___________________ self = def test_async_return_cmd_exc(self): py = create_tmp_test( """ import sys sys.exit(1) """ ) async def main(): await python(py.name, _async=True, _return_cmd=True) > self.assertRaises(sh.ErrorReturnCode_1, asyncio.run, main()) tests/sh_test.py:1831: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1829: in main await python(py.name, _async=True, _return_cmd=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________________ FunctionalTests.test_async __________________________ self = def test_async(self): py = create_tmp_test( """ import os import time time.sleep(0.5) print("hello") """ ) alternating = [] async def producer(q): alternating.append(1) msg = await python(py.name, _async=True) alternating.append(1) await q.put(msg.strip()) async def consumer(q): await asyncio.sleep(0.1) alternating.append(2) msg = await q.get() self.assertEqual(msg, "hello") alternating.append(2) async def main(): q = AQueue() await asyncio.gather(producer(q), consumer(q)) > asyncio.run(main()) tests/sh_test.py:1743: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.14/asyncio/runners.py:204: in run return runner.run(main) ^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/runners.py:127: in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/asyncio/base_events.py:719: in run_until_complete return future.result() ^^^^^^^^^^^^^^^ tests/sh_test.py:1741: in main await asyncio.gather(producer(q), consumer(q)) tests/sh_test.py:1728: in producer msg = await python(py.name, _async=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e74d0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpignyw24f'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': True, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_multiple_bakes ______________________ self = def test_multiple_bakes(self): py = create_tmp_test( """ import sys sys.stdout.write(str(sys.argv[1:])) """ ) > out = python.bake(py.name).bake("bake1").bake("bake2")() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1363: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp0z0iwj3q', 'bake1', 'bake2'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_exit_code_with_hasattr __________________ self = def test_exit_code_with_hasattr(self): from sh import ErrorReturnCode_3 py = create_tmp_test( """ exit(3) """ ) try: > out = python(py.name, _iter=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:336: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp83y1uegg'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_callable_interact ____________________ self = def test_callable_interact(self): py = create_tmp_test( """ import sys sys.stdout.write("line1") """ ) class Callable: def __init__(self): self.line = None def __call__(self, line): self.line = line cb = Callable() > python(py.name, _out=cb) tests/sh_test.py:2362: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmps6tm1wz3'], stdin = None stdout = .Callable object at 0x7fa3aa7e6cf0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_err_to_out ________________________ self = def test_err_to_out(self): py = create_tmp_test( """ import sys import os sys.stdout.write("stdout") sys.stdout.flush() sys.stderr.write("stderr") sys.stderr.flush() """ ) > stdout = pythons(py.name, _err_to_out=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1149: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmppxkulvva'], stdin = None, stdout = None stderr = -1 call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_iter_unicode _______________________ self = def test_iter_unicode(self): # issue https://github.com/amoffat/sh/issues/224 test_string = "\xe4\xbd\x95\xe4\xbd\x95\n" * 150 # len > buffer_s txt = create_tmp_test(test_string) > for line in sh.cat(txt.name, _iter=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1869: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/cat', '/tmp/tmp3c2k91p4'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_long_bool_option _____________________ self = def test_long_bool_option(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("-l", "--long-option", action="store_true", default=False, \ dest="long_option") options, args = parser.parse_args() print(options.long_option) """ ) > self.assertTrue(python(py.name, long_option=True).strip() == "True") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:883: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6cf0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpliwvrc5l', '--long-option'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_pass_fds _________________________ self = def test_pass_fds(self): # guarantee some extra fds in our parent process that don't close on exec. # we have to explicitly do this because at some point (I believe python 3.4), # python started being more stringent with closing fds to prevent security # vulnerabilities. python 2.7, for example, doesn't set CLOEXEC on # tempfile.TemporaryFile()s # # https://www.python.org/dev/peps/pep-0446/ tmp = [tempfile.TemporaryFile() for i in range(10)] for t in tmp: flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD) flags &= ~fcntl.FD_CLOEXEC fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) last_fd = tmp[-1].fileno() py = create_tmp_test( """ import os print(os.listdir("/dev/fd")) """ ) > out = python(py.name, _pass_fds=[last_fd]).strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:683: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aae178c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp7bmzfkeo'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_piped_generator _____________________ self = def test_piped_generator(self): import time py1 = create_tmp_test( """ import sys import os import time for letter in "andrew": time.sleep(0.6) print(letter) """ ) py2 = create_tmp_test( """ import sys import os import time while True: line = sys.stdin.readline() if not line: break print(line.strip().upper()) """ ) times = [] last_received = None letters = "" > for line in python( "-u", py2.name, _iter=True, _in=python("-u", py1.name, _piped="out") ): tests/sh_test.py:2014: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpo2kymcw9'] stdin = stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_partially_applied_callback ________________ self = def test_partially_applied_callback(self): from functools import partial py = create_tmp_test( """ for i in range(10): print(i) """ ) output = [] def fn(foo, line): output.append((foo, int(line.strip()))) log_line = partial(fn, "hello") > python(py.name, _out=log_line) tests/sh_test.py:3035: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp3yw1mcbj'], stdin = None stdout = functools.partial(.fn at 0x7fa3a99b4670>, 'hello') stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_number_arg ________________________ self = def test_number_arg(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() options, args = parser.parse_args() print(args[0]) """ ) > out = python(py.name, 3).strip() ^^^^^^^^^^^^^^^^^^ tests/sh_test.py:271: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp4hti1mdy', '3'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_raw_args _________________________ self = def test_raw_args(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("--long_option", action="store", default=None, dest="long_option1") parser.add_option("--long-option", action="store", default=None, dest="long_option2") options, args = parser.parse_args() if options.long_option1: print(options.long_option1.upper()) else: print(options.long_option2.upper()) """ ) self.assertEqual( > python(py.name, {"long_option": "underscore"}).strip(), "UNDERSCORE" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) tests/sh_test.py:972: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e4830> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpo4rkmnox', '--long_option=underscore'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_output_equivalence ____________________ self = def test_output_equivalence(self): from sh import whoami > iam1 = whoami() ^^^^^^^^ tests/sh_test.py:1395: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/whoami'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_fg_false _________________________ self = def test_fg_false(self): """https://github.com/amoffat/sh/issues/520""" py = create_tmp_test("print('hello')") buf = StringIO() > python(py.name, _fg=False, _out=buf) tests/sh_test.py:2097: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp3w5jnz1k'], stdin = None stdout = <_io.StringIO object at 0x7fa3aaa33520>, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_stringio_input ______________________ self = def test_stringio_input(self): from sh import cat input = StringIO() input.write("herpderp") input.seek(0) > out = cat(_in=input) ^^^^^^^^^^^^^^ tests/sh_test.py:2283: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/cat'], stdin = <_io.StringIO object at 0x7fa3aaa33250> stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_internal_bufsize _____________________ self = def test_internal_bufsize(self): from sh import cat > output = cat(_in="a" * 1000, _internal_bufsize=100, _out_bufsize=0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2289: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/cat'] stdin = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_arg_preprocessor _____________________ self = def test_arg_preprocessor(self): py = create_tmp_test( """ import sys sys.stdout.write(str(sys.argv[1:])) """ ) def arg_preprocess(args, kwargs): args.insert(0, "preprocessed") kwargs["a-kwarg"] = 123 return args, kwargs cmd = pythons.bake(py.name, _arg_preprocess=arg_preprocess) > out = cmd("arg") ^^^^^^^^^^ tests/sh_test.py:1380: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp19z1mmx2', 'preprocessed', 'arg', '--a-kwarg=123'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': .arg_preprocess at 0x7fa3a99b4eb0>, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_manual_stdin_queue ____________________ self = def test_manual_stdin_queue(self): from sh import tr try: from Queue import Queue except ImportError: from queue import Queue test = ["testing\n", "herp\n", "derp\n"] q = Queue() for t in test: q.put(t) q.put(None) # EOF > out = tr("[:lower:]", "[:upper:]", _in=q) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:536: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3ac2786e0> command = , parent_log = cmd = ['/usr/bin/tr', '[:lower:]', '[:upper:]'] stdin = , stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________ FunctionalTests.test_ok_code_ignores_bad_sig_exception ____________ self = def test_ok_code_ignores_bad_sig_exception(self): # Test if I have [-sig] in _ok_code, the exception won't be raised py = create_tmp_test( """ import time import sys time.sleep(2) sys.exit(1) """ ) for sig in SIGNALS_THAT_SHOULD_THROW_EXCEPTION: if sig == signal.SIGPIPE: continue sig_exception_name = f"SignalException_{sig}" sig_exception = getattr(sh, sig_exception_name) python_bg_no_sig_exception = python_bg.bake(_ok_code=[-sig]) try: > p = python_bg_no_sig_exception(py.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:3199: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpa88o6epz'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_cwd_fg __________________________ self = def test_cwd_fg(self): td = realpath(tempfile.mkdtemp()) py = create_tmp_test( f""" import sh import os from os.path import realpath orig = realpath(os.getcwd()) print(orig) sh.pwd(_cwd="{td}", _fg=True) print(realpath(os.getcwd())) """ ) > orig, newdir, restored = python(py.name).strip().split("\n") ^^^^^^^^^^^^^^^ tests/sh_test.py:2176: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e46e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpzcobmik3'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_binary_pipe _______________________ self = def test_binary_pipe(self): binary = b"\xec;\xedr\xdbF" py1 = create_tmp_test( """ import sys import os sys.stdout = os.fdopen(sys.stdout.fileno(), "wb", 0) sys.stdout.write(b'\\xec;\\xedr\\xdbF') """ ) py2 = create_tmp_test( """ import sys import os sys.stdin = os.fdopen(sys.stdin.fileno(), "rb", 0) sys.stdout = os.fdopen(sys.stdout.fileno(), "wb", 0) sys.stdout.write(sys.stdin.read()) """ ) > out = python(py2.name, _in=python(py1.name)) ^^^^^^^^^^^^^^^^ tests/sh_test.py:2429: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e74d0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp72pb8rx3'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_err_redirection_actual_file _______________ self = def test_err_redirection_actual_file(self): import tempfile file_obj = tempfile.NamedTemporaryFile() py = create_tmp_test( """ import sys import os sys.stdout.write("stdout") sys.stderr.write("stderr") """ ) > stdout = pythons("-u", py.name, _err=file_obj.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1329: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpb7oic432'], stdin = None, stdout = None stderr = <_io.BufferedWriter name='/tmp/tmp659mwg6i'> call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_unicode_exception ____________________ self = def test_unicode_exception(self): from sh import ErrorReturnCode py = create_tmp_test("exit(1)") arg = "漢字" native_arg = arg try: > python(py.name, arg, _encoding="utf8") tests/sh_test.py:237: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e46e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmppuuiewbh', '漢字'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_signal_exception _____________________ self = def test_signal_exception(self): from sh import SignalException_15 def throw_terminate_signal(): py = create_tmp_test( """ import time while True: time.sleep(1) """ ) to_kill = python(py.name, _bg=True) to_kill.terminate() to_kill.wait() > self.assertRaises(SignalException_15, throw_terminate_signal) tests/sh_test.py:2575: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:2571: in throw_terminate_signal to_kill = python(py.name, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_failure_with_large_output ________________ self = def test_failure_with_large_output(self): from sh import ErrorReturnCode_1 py = create_tmp_test( """ print("andrewmoffat" * 1000) exit(1) """ ) > self.assertRaises(ErrorReturnCode_1, python, py.name) tests/sh_test.py:2443: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_arg_string_coercion ___________________ self = def test_arg_string_coercion(self): py = create_tmp_test( """ from argparse import ArgumentParser parser = ArgumentParser() parser.add_argument("-n", type=int) parser.add_argument("--number", type=int) ns = parser.parse_args() print(ns.n + ns.number) """ ) > out = python(py.name, n=3, number=4, _long_sep=None).strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:286: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpv5tu645j', '-n', '3', '--number', '4'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_stdout_callback_buffered _________________ self = def test_stdout_callback_buffered(self): py = create_tmp_test( """ import sys import os for i in range(5): sys.stdout.write("herpderp") """ ) stdout = [] def agg(chunk): stdout.append(chunk) > p = python("-u", py.name, _out=agg, _out_bufsize=4) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1529: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpxjrdd9td'], stdin = None stdout = .agg at 0x7fa3a82a4ca0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_no_proc_no_attr _____________________ self = def test_no_proc_no_attr(self): py = create_tmp_test("") > with python(py.name) as p: ^^^^^^^^^^^^^^^ tests/sh_test.py:3016: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aae178c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpy1_fw204'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_stdin_newline_bufsize __________________ self = def test_stdin_newline_bufsize(self): from time import sleep # this tries to receive some known data and measures the time it takes # to receive it. since we're flushing by newline, we should only be # able to receive the data when a newline is fed in py = create_tmp_test( """ import sys from time import time started = time() data = sys.stdin.read(len("testing\\n")) waited = time() - started sys.stdout.write(data) sys.stdout.write(str(waited) + "\\n") started = time() data = sys.stdin.read(len("done\\n")) waited = time() - started sys.stdout.write(data) sys.stdout.write(str(waited) + "\\n") sys.stdout.flush() """ ) # we'll feed in text incrementally, sleeping strategically before # sending a newline. we then measure the amount that we slept # indirectly in the child process def create_stdin(): yield "test" sleep(1) yield "ing\n" sleep(1) yield "done\n" > out = python(py.name, _in=create_stdin(), _in_bufsize=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2962: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpdl62le4m'] stdin = .create_stdin at 0x7fa3a842be00> stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_bg_exit_code _______________________ self = def test_bg_exit_code(self): py = create_tmp_test( """ import time time.sleep(1) exit(49) """ ) > p = python(py.name, _ok_code=49, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2151: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpzfffqjna'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_tty_output ________________________ self = def test_tty_output(self): py = create_tmp_test( """ import sys import os if os.isatty(sys.stdout.fileno()): sys.stdout.write("tty attached") sys.stdout.flush() else: sys.stdout.write("no tty attached") sys.stdout.flush() """ ) > out = pythons(py.name, _tty_out=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2252: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpl8ctj497'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_manual_stdin_file ____________________ self = def test_manual_stdin_file(self): import tempfile from sh import tr test_string = "testing\nherp\nderp\n" stdin = tempfile.NamedTemporaryFile() stdin.write(test_string.encode()) stdin.flush() stdin.seek(0) > out = tr("[:lower:]", "[:upper:]", _in=stdin) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:517: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/tr', '[:lower:]', '[:upper:]'] stdin = <_TemporaryFileWrapper file=<_io.BufferedRandom name='/tmp/tmpxi06eu_i'>> stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_shadowed_subcommand ___________________ self = def test_shadowed_subcommand(self): py = create_tmp_test( """ import sys sys.stdout.write(sys.argv[1]) """ ) > out = pythons.bake(py.name).bake_() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:3011: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpezsk7msc', 'bake'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_general_signal ______________________ self = def test_general_signal(self): from signal import SIGINT py = create_tmp_test( """ import sys import os import time import signal i = 0 def sig_handler(sig, frame): global i i = 42 signal.signal(signal.SIGINT, sig_handler) for _ in range(6): print(i) i += 1 sys.stdout.flush() time.sleep(2) """ ) stdout = [] def agg(line, stdin, process): line = line.strip() stdout.append(line) if line == "3": process.signal(SIGINT) return True > p = python(py.name, _out=agg, _tee=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1689: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e5550> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp4fjpsh5j'], stdin = None stdout = .agg at 0x7fa3a82a6980> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_multiple_args_long_option ________________ self = def test_multiple_args_long_option(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("-l", "--long-option", dest="long_option") options, args = parser.parse_args() print(len(options.long_option.split())) """ ) > num_args = int(python(py.name, long_option="one two three", nothing=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:852: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6cf0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpo63iskxt', '--long-option=one two three'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_composition _______________________ self = def test_composition(self): py1 = create_tmp_test( """ import sys print(int(sys.argv[1]) * 2) """ ) py2 = create_tmp_test( """ import sys print(int(sys.argv[1]) + 1) """ ) > res = python(py2.name, python(py1.name, 8)).strip() ^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:913: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e4830> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpcgxa2o52', '8'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_exit_code_from_exception _________________ self = def test_exit_code_from_exception(self): from sh import ErrorReturnCode_3 py = create_tmp_test( """ exit(3) """ ) > self.assertRaises(ErrorReturnCode_3, python, py.name) tests/sh_test.py:354: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_empty_stdin_no_hang ___________________ self = def test_empty_stdin_no_hang(self): py = create_tmp_test( """ import sys data = sys.stdin.read() sys.stdout.write("no hang") """ ) > out = pythons(py.name, _in="", _timeout=2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:297: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpmwtgt1k7'], stdin = '', stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_cast_bg _________________________ self = def test_cast_bg(self): py = create_tmp_test( """ import sys import time time.sleep(0.5) sys.stdout.write(sys.argv[1]) """ ) > self.assertEqual(int(python(py.name, "123", _bg=True)), 123) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2073: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3ac2786e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpzr6hi3rp', '123'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_no_out_iter_err _____________________ self = def test_no_out_iter_err(self): py = create_tmp_test( """ import sys sys.stderr.write("1\\n") sys.stderr.write("2\\n") sys.stderr.write("3\\n") sys.stderr.flush() """ ) > nums = [int(num.strip()) for num in python(py.name, _iter="err", _no_out=True)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2037: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpbv9y2_wd'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -2 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_background ________________________ self = def test_background(self): import time from sh import sleep start = time.time() sleep_time = 0.5 > p = sleep(sleep_time, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1054: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/sleep', '0.5'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_stdout_callback_with_input ________________ self = def test_stdout_callback_with_input(self): py = create_tmp_test( """ import sys import os for i in range(5): print(str(i)) derp = input("herp? ") print(derp) """ ) def agg(line, stdin): if line.strip() == "4": stdin.put("derp\n") > p = python("-u", py.name, _out=agg, _tee=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1550: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp12w7gmym'], stdin = None stdout = .agg at 0x7fa3a81f9430> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _____________________ FunctionalTests.test_command_wrapper _____________________ self = def test_command_wrapper(self): from sh import Command, which > ls = Command(str(which("ls")).strip()) ^^^^^^^^^^^ tests/sh_test.py:1039: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e46e0> command = , parent_log = cmd = ['/usr/bin/which', 'ls'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________ FunctionalTests.test_handle_both_out_and_err _________________ self = def test_handle_both_out_and_err(self): py = create_tmp_test( """ import sys import os import time for i in range(42): sys.stdout.write(str(i) + "\\n") sys.stdout.flush() if i % 2 == 0: sys.stderr.write(str(i) + "\\n") sys.stderr.flush() """ ) out = [] def handle_out(line): out.append(int(line.strip())) err = [] def handle_err(line): err.append(int(line.strip())) > p = python(py.name, _err=handle_err, _out=handle_out, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1859: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmphpatflnc'], stdin = None stdout = .handle_out at 0x7fa3a82a5d20> stderr = .handle_err at 0x7fa3a82a57a0> call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ FunctionalTests.test_no_out __________________________ self = def test_no_out(self): py = create_tmp_test( """ import sys sys.stdout.write("stdout") sys.stderr.write("stderr") """ ) > p = python(py.name, _no_out=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2461: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpuzqnn6mz'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_stdout_callback_kill ___________________ self = def test_stdout_callback_kill(self): import signal py = create_tmp_test( """ import sys import os import time for i in range(5): print(i) time.sleep(.5) """ ) stdout = [] def agg(line, stdin, process): line = line.strip() stdout.append(line) if line == "3": process.kill() return True import sh caught_signal = False try: > p = python("-u", py.name, _out=agg, _bg=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1645: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpkpalns9b'], stdin = None stdout = .agg at 0x7fa3a81f8300> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_short_bool_option ____________________ self = def test_short_bool_option(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("-s", action="store_true", default=False, dest="short_option") options, args = parser.parse_args() print(options.short_option) """ ) > self.assertTrue(python(py.name, s=True).strip() == "True") ^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:868: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmprr3b5_tn', '-s'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_bake_args_come_first ___________________ self = def test_bake_args_come_first(self): from sh import ls ls = ls.bake(h=True) > ran = ls("-la", _return_cmd=True).ran ^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1388: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e74d0> command = , parent_log = cmd = ['/usr/bin/ls', '-h', '-la'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_with_context_args ____________________ self = def test_with_context_args(self): import getpass from sh import whoami py = create_tmp_test( """ import sys import os import subprocess from optparse import OptionParser parser = OptionParser() parser.add_option("-o", "--opt", action="store_true", default=False, dest="opt") options, args = parser.parse_args() if options.opt: subprocess.Popen(args[0], shell=False).wait() """ ) with python(py.name, opt=True, _with=True): > out = whoami() ^^^^^^^^ tests/sh_test.py:1111: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5a90> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpasfe7c7t', '--opt', '/usr/bin/whoami'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_tty_stdin ________________________ self = def test_tty_stdin(self): py = create_tmp_test( """ import sys sys.stdout.write(sys.stdin.read()) sys.stdout.flush() """ ) > out = pythons(py.name, _in="test\n", _tty_in=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2487: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'pid'") raised in repr()] OProc object at 0x7fa3aae178c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp5q4vfz38'], stdin = 'test\n', stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: > self.ctty = os.ttyname(self._stdin_child_fd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E OSError: [Errno 25] Inappropriate ioctl for device sh.py:1963: OSError _______________________ FunctionalTests.test_long_option _______________________ self = def test_long_option(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("-l", "--long-option", action="store", default="", dest="long_option") options, args = parser.parse_args() print(options.long_option.upper()) """ ) > self.assertTrue(python(py.name, long_option="testing").strip() == "TESTING") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:951: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6cf0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp4qlgy70a', '--long-option=testing'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_with_context _______________________ self = def test_with_context(self): import getpass from sh import whoami py = create_tmp_test( """ import sys import os import subprocess print("with_context") subprocess.Popen(sys.argv[1:], shell=False).wait() """ ) cmd1 = python.bake(py.name, _with=True) with cmd1: > out = whoami() ^^^^^^^^ tests/sh_test.py:1086: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp6mtg8q1y', '/usr/bin/whoami'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_manual_stdin_string ___________________ self = def test_manual_stdin_string(self): from sh import tr > out = tr("[:lower:]", "[:upper:]", _in="andrew").strip() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:493: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e70e0> command = , parent_log = cmd = ['/usr/bin/tr', '[:lower:]', '[:upper:]'], stdin = 'andrew', stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_background_exception ___________________ self = def test_background_exception(self): py = create_tmp_test("exit(1)") > p = python(py.name, _bg=True, _bg_exc=False) # should not raise ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1065: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7620> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp9ybo27nx'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': False, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ FunctionalTests.test_nonblocking_iter _____________________ self = def test_nonblocking_iter(self): from errno import EWOULDBLOCK py = create_tmp_test( """ import time import sys time.sleep(1) sys.stdout.write("stdout") """ ) count = 0 value = None > for line in python(py.name, _iter_noblock=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1886: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e6900> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpws0715fo'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_for_generator_to_err ___________________ self = def test_for_generator_to_err(self): py = create_tmp_test( """ import sys import os for i in range(42): sys.stderr.write(str(i)+"\\n") """ ) out = [] > for line in python("-u", py.name, _iter="err"): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1925: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e5550> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpy6yk2g_u'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -2 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________ FunctionalTests.test_done_callback_no_deadlock ________________ self = def test_done_callback_no_deadlock(self): import time py = create_tmp_test( """ from sh import sleep def done(cmd, success, exit_code): print(cmd, success, exit_code) sleep('1', _done=done) """ ) > p = python(py.name, _bg=True, _timeout=2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:2741: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e78c0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpi_avx0_8'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': True, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________ FunctionalTests.test_subcommand_and_bake ___________________ self = def test_subcommand_and_bake(self): import getpass py = create_tmp_test( """ import sys import os import subprocess print("subcommand") subprocess.Popen(sys.argv[1:], shell=False).wait() """ ) cmd1 = python.bake(py.name) > out = cmd1.whoami() ^^^^^^^^^^^^^ tests/sh_test.py:1351: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b6270> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp3aekv6zf', 'whoami'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ FunctionalTests.test_decode_error_handling __________________ self = def test_decode_error_handling(self): from functools import partial py = create_tmp_test( """ # -*- coding: utf8 -*- import sys import os sys.stdout = os.fdopen(sys.stdout.fileno(), 'wb') sys.stdout.write(bytes("te漢字st", "utf8") + "äåéë".encode("latin_1")) """ ) fn = partial(pythons, py.name, _encoding="ascii") > self.assertRaises(UnicodeDecodeError, fn) tests/sh_test.py:2547: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________ FunctionalTests.test_multiple_args_short_option ________________ self = def test_multiple_args_short_option(self): py = create_tmp_test( """ from optparse import OptionParser parser = OptionParser() parser.add_option("-l", dest="long_option") options, args = parser.parse_args() print(len(options.long_option.split())) """ ) > num_args = int(python(py.name, l="one two three")) # noqa: E741 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:836: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7cb0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpeibn8f92', '-l', 'one two three'] stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_err_piped ________________________ self = def test_err_piped(self): py = create_tmp_test( """ import sys sys.stderr.write("stderr") """ ) py2 = create_tmp_test( """ import sys while True: line = sys.stdin.read() if not line: break sys.stdout.write(line) """ ) > out = pythons("-u", py2.name, _in=python("-u", py.name, _piped="err")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1188: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e74d0> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmp8iaxjcj2'] stdin = stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_trunc_exc ________________________ self = def test_trunc_exc(self): py = create_tmp_test( """ import sys sys.stdout.write("a" * 1000) sys.stderr.write("b" * 1000) exit(1) """ ) > self.assertRaises(sh.ErrorReturnCode_1, python, py.name) tests/sh_test.py:259: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________ FunctionalTests.test_stdout_callback_line_buffered ______________ self = def test_stdout_callback_line_buffered(self): py = create_tmp_test( """ import sys import os for i in range(5): print("herpderp") """ ) stdout = [] def agg(line): stdout.append(line) > p = python("-u", py.name, _out=agg, _out_bufsize=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:1488: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7380> command = , parent_log = cmd = ['/usr/bin/python', '-u', '/tmp/tmpxdspnqzk'], stdin = None stdout = .agg at 0x7fa3a97e45c0> stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ FunctionalTests.test_print_command ______________________ self = def test_print_command(self): from sh import ls, which > actual_location = which("ls").strip() ^^^^^^^^^^^ tests/sh_test.py:216: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e57f0> command = , parent_log = cmd = ['/usr/bin/which', 'ls'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _______________________ FunctionalTests.test_unicode_arg _______________________ self = def test_unicode_arg(self): from sh import echo test = "漢字" > p = echo(test, _encoding="utf8") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:224: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7a10> command = , parent_log = cmd = ['/usr/bin/echo', '漢字'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ________________________ FunctionalTests.test_exception ________________________ self = def test_exception(self): from sh import ErrorReturnCode_2 py = create_tmp_test( """ exit(2) """ ) > self.assertRaises(ErrorReturnCode_2, python, py.name) tests/sh_test.py:721: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException __________________ MiscTests.test_percent_doesnt_fail_logging __________________ self = def test_percent_doesnt_fail_logging(self): """test that a command name doesn't interfere with string formatting in the internal loggers""" py = create_tmp_test( """ print("cool") """ ) > python(py.name, "%") tests/sh_test.py:3290: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b6120> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmp8pjsnki3', '%'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ______________________ MiscTests.test_change_log_message _______________________ self = def test_change_log_message(self): py = create_tmp_test( """ print("cool") """ ) def log_msg(cmd, call_args, pid=None): return "Hi! I ran something" buf = StringIO() handler = logging.StreamHandler(buf) logger = logging.getLogger("sh") logger.setLevel(logging.INFO) try: logger.addHandler(handler) > python(py.name, "meow", "bark", _log_msg=log_msg) tests/sh_test.py:3406: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e7b60> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpu25edm0u', 'meow', 'bark'], stdin = None stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ MiscTests.test_stdin_nohang __________________________ self = def test_stdin_nohang(self): py = create_tmp_test( """ print("hi") """ ) read, write = os.pipe() stdin = os.fdopen(read, "r") > python(py.name, _in=stdin) tests/sh_test.py:3336: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e46e0> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpx7930_au'] stdin = <_io.TextIOWrapper name=678 mode='r' encoding='UTF-8'>, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException _________________________ MiscTests.test_unicode_path __________________________ self = @requires_utf8 def test_unicode_path(self): from sh import Command python_name = os.path.basename(sys.executable) py = create_tmp_test( f"""#!/usr/bin/env {python_name} # -*- coding: utf8 -*- print("字") """, prefix="字", delete=False, ) try: py.close() os.chmod(py.name, int(0o755)) cmd = Command(py.name) # all of these should behave just fine str(cmd) repr(cmd) > running = cmd(_return_cmd=True) ^^^^^^^^^^^^^^^^^^^^^ tests/sh_test.py:3361: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3aa7e4830> command = , parent_log = cmd = ['/tmp/字73196x2v'], stdin = None, stdout = None, stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ____________________ MiscTests.test_threaded_with_contexts _____________________ self = def test_threaded_with_contexts(self): import threading import time py = create_tmp_test( """ import sys a = sys.argv res = (a[1], a[3]) sys.stdout.write(repr(res)) """ ) p1 = python.bake("-u", py.name, 1) p2 = python.bake("-u", py.name, 2) results = [None, None] def f1(): with p1: time.sleep(1) results[0] = str(system_python("one")) def f2(): with p2: results[1] = str(system_python("two")) t1 = threading.Thread(target=f1) t1.start() t2 = threading.Thread(target=f2) t2.start() t1.join() t2.join() correct = [ "('1', 'one')", "('2', 'two')", ] > self.assertEqual(results, correct) E AssertionError: Lists differ: [None, None] != ["('1', 'one')", "('2', 'two')"] E E First differing element 0: E None E "('1', 'one')" E E - [None, None] E + ["('1', 'one')", "('2', 'two')"] tests/sh_test.py:3470: AssertionError _________________________ MiscTests.test_fd_over_1024 __________________________ self = @requires_poller("poll") def test_fd_over_1024(self): py = create_tmp_test("""print("hi world")""") with ulimit(resource.RLIMIT_NOFILE, 2048): cutoff_fd = 1024 pipes = [] for i in range(cutoff_fd): master, slave = os.pipe() pipes.append((master, slave)) if slave >= cutoff_fd: break > python(py.name) tests/sh_test.py:3274: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'cmd'") raised in repr()] OProc object at 0x7fa3a94b5d30> command = , parent_log = cmd = ['/usr/bin/python', '/tmp/tmpcdr2k5x_'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: self._stdin_child_fd, self._stdin_parent_fd = os.pipe() if stdout_is_fd_based and not tee_out: self._stdout_child_fd = os.dup(get_fileno(stdout)) self._stdout_parent_fd = None # tty_out=True is the default elif ca["tty_out"]: self._stdout_parent_fd, self._stdout_child_fd = pty.openpty() else: self._stdout_parent_fd, self._stdout_child_fd = os.pipe() # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe, # and never a PTY. the reason for this is not totally clear to me, # but it has to do with the fact that if STDERR isn't set as the # CTTY (because STDOUT is), the STDERR buffer won't always flush # by the time the process exits, and the data will be lost. # i've only seen this on OSX. if stderr is OProc.STDOUT: # if stderr is going to stdout, but stdout is a tty or a pipe, # we should not specify a read_fd, because stdout is os.dup'ed # directly to the stdout fd (no pipe), and so stderr won't have # a slave end of a pipe either to dup if stdout_is_fd_based and not tee_out: self._stderr_parent_fd = None else: self._stderr_parent_fd = os.dup(self._stdout_parent_fd) self._stderr_child_fd = os.dup(self._stdout_child_fd) elif stderr_is_fd_based and not tee_err: self._stderr_child_fd = os.dup(get_fileno(stderr)) self._stderr_parent_fd = None else: self._stderr_parent_fd, self._stderr_child_fd = os.pipe() piped = ca["piped"] self._pipe_fd = None if piped: fd_to_use = self._stdout_parent_fd if piped == "err": fd_to_use = self._stderr_parent_fd self._pipe_fd = os.dup(fd_to_use) new_session = ca["new_session"] new_group = ca["new_group"] needs_ctty = ca["tty_in"] # if we need a controlling terminal, we have to be in a new session where we # are the session leader, otherwise we would need to take over the existing # process session, and we can't do that(?) if needs_ctty: new_session = True self.ctty = None if needs_ctty: self.ctty = os.ttyname(self._stdin_child_fd) gc_enabled = gc.isenabled() if gc_enabled: gc.disable() # for synchronizing session_pipe_read, session_pipe_write = os.pipe() exc_pipe_read, exc_pipe_write = os.pipe() # this pipe is for synchronizing with the child that the parent has # closed its in/out/err fds. this is a bug on OSX (but not linux), # where we can lose output sometimes, due to a race, if we do # os.close(self._stdout_child_fd) in the parent after the child starts # writing. if IS_MACOS: close_pipe_read, close_pipe_write = os.pipe() else: close_pipe_read, close_pipe_write = None, None # session id, group id, process id self.sid = None self.pgid = None self.pid = os.fork() # child if self.pid == 0: # pragma: no cover if IS_MACOS: os.read(close_pipe_read, 1) os.close(close_pipe_read) os.close(close_pipe_write) # this is critical # our exc_pipe_write must have CLOEXEC enabled. the reason for this is # tricky: if our child (the block we're in now), has an exception, we need # to be able to write to exc_pipe_write, so that when the parent does # os.read(exc_pipe_read), it gets our traceback. however, # os.read(exc_pipe_read) in the parent blocks, so if our child *doesn't* # have an exception, and doesn't close the writing end, it hangs forever. # not good! but obviously the child can't close the writing end until it # knows it's not going to have an exception, which is impossible to know # because but what if os.execv has an exception? so the answer is CLOEXEC, # so that the writing end of the pipe gets closed upon successful exec, # and the parent reading the read end won't block (close breaks the block). flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD) flags |= fcntl.FD_CLOEXEC fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) try: # ignoring SIGHUP lets us persist even after the controlling terminal # is closed if ca["bg"] is True: signal.signal(signal.SIGHUP, signal.SIG_IGN) # python ignores SIGPIPE by default. we must make sure to put # this behavior back to the default for spawned processes, # otherwise SIGPIPE won't kill piped processes, which is what we # need, so that we can check the error code of the killed # process to see that SIGPIPE killed it signal.signal(signal.SIGPIPE, signal.SIG_DFL) # put our forked process in a new session? this will relinquish # any control of our inherited CTTY and also make our parent # process init if new_session: os.setsid() elif new_group: os.setpgid(0, 0) sid = os.getsid(0) pgid = os.getpgid(0) payload = (f"{sid},{pgid}").encode(DEFAULT_ENCODING) os.write(session_pipe_write, payload) if ca["tty_out"] and not stdout_is_fd_based and not single_tty: # set raw mode, so there isn't any weird translation of # newlines to \r\n and other oddities. we're not outputting # to a terminal anyways # # we HAVE to do this here, and not in the parent process, # because we have to guarantee that this is set before the # child process is run, and we can't do it twice. tty.setraw(self._stdout_child_fd) # if the parent-side fd for stdin exists, close it. the case # where it may not exist is if we're using piping if self._stdin_parent_fd: os.close(self._stdin_parent_fd) if self._stdout_parent_fd: os.close(self._stdout_parent_fd) if self._stderr_parent_fd: os.close(self._stderr_parent_fd) os.close(session_pipe_read) os.close(exc_pipe_read) cwd = ca["cwd"] if cwd: os.chdir(cwd) os.dup2(self._stdin_child_fd, 0) os.dup2(self._stdout_child_fd, 1) os.dup2(self._stderr_child_fd, 2) # set our controlling terminal, but only if we're using a tty # for stdin. it doesn't make sense to have a ctty otherwise if needs_ctty: tmp_fd = os.open(os.ttyname(0), os.O_RDWR) os.close(tmp_fd) if ca["tty_out"] and not stdout_is_fd_based: setwinsize(1, ca["tty_size"]) if ca["uid"] is not None: os.setgid(target_gid) os.setuid(target_uid) preexec_fn = ca["preexec_fn"] if callable(preexec_fn): preexec_fn() close_fds = ca["close_fds"] if ca["pass_fds"]: close_fds = True if close_fds: pass_fds = {0, 1, 2, exc_pipe_write} pass_fds.update(ca["pass_fds"]) # don't inherit file descriptors try: inherited_fds = os.listdir("/dev/fd") except OSError: # Some systems don't have /dev/fd. Raises OSError in # Python2, FileNotFoundError on Python3. The latter doesn't # exist on Python2, but inherits from IOError, which does. inherited_fds = os.listdir("/proc/self/fd") inherited_fds = {int(fd) for fd in inherited_fds} - pass_fds for fd in inherited_fds: try: os.close(fd) except OSError: pass # python=3.6, locale=c will fail test_unicode_arg if we don't # explicitly encode to bytes via our desired encoding. this does # not seem to be the case in other python versions, even if locale=c bytes_cmd = [c.encode(ca["encoding"]) for c in cmd] # actually execute the process if ca["env"] is None: os.execv(bytes_cmd[0], bytes_cmd) else: os.execve(bytes_cmd[0], bytes_cmd, ca["env"]) # we must ensure that we carefully exit the child process on # exception, otherwise the parent process code will be executed # twice on exception https://github.com/amoffat/sh/issues/202 # # if your parent process experiences an exit code 255, it is most # likely that an exception occurred between the fork of the child # and the exec. this should be reported. except Exception: # noqa: E722 # some helpful debugging tb = traceback.format_exc().encode("utf8", "ignore") try: os.write(exc_pipe_write, tb) except Exception as e: # dump to stderr if we cannot save it to exc_pipe_write sys.stderr.write(f"\nFATAL SH ERROR: {e}\n") finally: os._exit(255) # parent else: if gc_enabled: gc.enable() os.close(self._stdin_child_fd) os.close(self._stdout_child_fd) os.close(self._stderr_child_fd) # tell our child process that we've closed our write_fds, so it is # ok to proceed towards exec. see the comment where this pipe is # opened, for why this is necessary if IS_MACOS: os.close(close_pipe_read) os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING)) os.close(close_pipe_write) os.close(exc_pipe_write) fork_exc = os.read(exc_pipe_read, 1024**2) os.close(exc_pipe_read) if fork_exc: fork_exc = fork_exc.decode(DEFAULT_ENCODING) > raise ForkException(fork_exc) E sh.ForkException: E E Original exception: E =================== E E Traceback (most recent call last): E File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ E tty.setraw(self._stdout_child_fd) E ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ E File "/usr/lib/python3.14/tty.py", line 61, in setraw E mode = tcgetattr(fd) E termios.error: (25, 'Inappropriate ioctl for device') sh.py:2163: ForkException ___________________________ MiscTests.test_pickling ____________________________ self = def test_pickling(self): import pickle > py = create_tmp_test( """ import sys sys.stdout.write("some output") sys.stderr.write("some error") exit(1) """ ) tests/sh_test.py:3242: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmpcw084xp8' /usr/lib/python3.14/tempfile.py:257: OSError _____________________________ MiscTests.test_eintr _____________________________ self = def test_eintr(self): import signal def handler(num, frame): pass signal.signal(signal.SIGALRM, handler) > py = create_tmp_test( """ import time time.sleep(2) """ ) tests/sh_test.py:3481: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmp1ct0o2e2' /usr/lib/python3.14/tempfile.py:257: OSError __________________ MiscTests.test_stop_iteration_doesnt_block __________________ self = def test_stop_iteration_doesnt_block(self): """proves that calling calling next() on a stopped iterator doesn't hang.""" > py = create_tmp_test( """ print("cool") """ ) tests/sh_test.py:3418: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmp4r5bmio1' /usr/lib/python3.14/tempfile.py:257: OSError ___________________ ExecutionContextTests.test_no_interfere1 ___________________ self = def test_no_interfere1(self): import sh > py = create_tmp_test( """ import sys sys.stdout.write(sys.argv[1]) """ ) tests/sh_test.py:3558: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmp4fqe918f' /usr/lib/python3.14/tempfile.py:257: OSError _______________________ ExecutionContextTests.test_basic _______________________ self = def test_basic(self): import sh > py = create_tmp_test( """ import sys sys.stdout.write(sys.argv[1]) """ ) tests/sh_test.py:3527: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmpexmhk4in' /usr/lib/python3.14/tempfile.py:257: OSError ________________ ExecutionContextTests.test_multiline_defaults _________________ self = def test_multiline_defaults(self): > py = create_tmp_test( """ import os print(os.environ["ABC"]) """ ) tests/sh_test.py:3540: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmp9ucc9nrm' /usr/lib/python3.14/tempfile.py:257: OSError ______________ ExecutionContextTests.test_set_in_parent_function _______________ self = def test_set_in_parent_function(self): import sh > py = create_tmp_test( """ import sys sys.stdout.write(sys.argv[1]) """ ) tests/sh_test.py:3591: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/sh_test.py:150: in create_tmp_test py = tempfile.NamedTemporaryFile(prefix=prefix, delete=delete) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /usr/lib/python3.14/tempfile.py:603: in NamedTemporaryFile ??? /usr/lib/python3.14/tempfile.py:600: in opener ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/tmp', pre = 'tmp', suf = '', flags = 131266, output_type = > ??? E OSError: [Errno 24] Too many open files: '/tmp/tmpk3b88tpd' /usr/lib/python3.14/tempfile.py:257: OSError ___________________ ExecutionContextTests.test_no_interfere2 ___________________ self = def test_no_interfere2(self): import sh out = StringIO() from sh import echo _sh = sh.bake(_out=out) # noqa: F841 > echo("-n", "TEST") tests/sh_test.py:3585: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sh.py:1511: in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sh.py:717: in __init__ self.process = OProc( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'OProc' object has no attribute 'pid'") raised in repr()] OProc object at 0x7fa3a94b5a90> command = , parent_log = cmd = ['/usr/bin/echo', '-n', 'TEST'], stdin = None, stdout = None stderr = None call_args = {'arg_preprocess': None, 'async': False, 'bg': False, 'bg_exc': True, ...} pipe = -1 process_assign_lock = def __init__( self, command, parent_log, cmd, stdin, stdout, stderr, call_args, pipe, process_assign_lock, ): """ cmd is the full list of arguments that will be exec'd. it includes the program name and all its arguments. stdin, stdout, stderr are what the child will use for standard input/output/err. call_args is a mapping of all the special keyword arguments to apply to the child process. """ self.command = command self.call_args = call_args # convenience ca = self.call_args if ca["uid"] is not None: if os.getuid() != 0: raise RuntimeError("UID setting requires root privileges") target_uid = ca["uid"] pwrec = pwd.getpwuid(ca["uid"]) target_gid = pwrec.pw_gid else: target_uid, target_gid = None, None # I had issues with getting 'Input/Output error reading stdin' from dd, # until I set _tty_out=False if ca["piped"]: ca["tty_out"] = False self._stdin_process = None # if the objects that we are passing to the OProc happen to be a # file-like object that is a tty, for example `sys.stdin`, then, later # on in this constructor, we're going to skip out on setting up pipes # and pseudoterminals for those endpoints stdin_is_fd_based = ob_is_fd_based(stdin) stdout_is_fd_based = ob_is_fd_based(stdout) stderr_is_fd_based = ob_is_fd_based(stderr) if isinstance(ca["tee"], (str, bool, int)) or ca["tee"] is None: tee = {ca["tee"]} else: tee = set(ca["tee"]) tee_out = TEE_STDOUT.intersection(tee) tee_err = TEE_STDERR.intersection(tee) single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"] # this logic is a little convoluted, but basically this top-level # if/else is for consolidating input and output TTYs into a single # TTY. this is the only way some secure programs like ssh will # output correctly (is if stdout and stdin are both the same TTY) if single_tty: # master_fd, slave_fd = pty.openpty() # # Anything that is written on the master end is provided to the process on # the slave end as though it was # input typed on a terminal. -"man 7 pty" # # later, in the child process, we're going to do this, so keep it in mind: # # os.dup2(self._stdin_child_fd, 0) # os.dup2(self._stdout_child_fd, 1) # os.dup2(self._stderr_child_fd, 2) self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # this makes our parent fds behave like a terminal. it says that the very # same fd that we "type" to (for stdin) is the same one that we see output # printed to (for stdout) self._stdout_parent_fd = os.dup(self._stdin_parent_fd) # this line is what makes stdout and stdin attached to the same pty. in # other words the process will write to the same underlying fd as stdout # as it uses to read from for stdin. this makes programs like ssh happy self._stdout_child_fd = os.dup(self._stdin_child_fd) self._stderr_parent_fd = os.dup(self._stdin_parent_fd) self._stderr_child_fd = os.dup(self._stdin_child_fd) # do not consolidate stdin and stdout. this is the most common use- # case else: # this check here is because we may be doing piping and so our stdin # might be an instance of OProc if isinstance(stdin, OProc) and stdin.call_args["piped"]: self._stdin_child_fd = stdin._pipe_fd self._stdin_parent_fd = None self._stdin_process = stdin elif stdin_is_fd_based: self._stdin_child_fd = os.dup(get_fileno(stdin)) self._stdin_parent_fd = None elif ca["tty_in"]: self._stdin_parent_fd, self._stdin_child_fd = pty.openpty() # tty_in=False is the default else: > self._stdin_child_fd, self._stdin_parent_fd = os.pipe() ^^^^^^^^^ E OSError: [Errno 24] Too many open files sh.py:1906: OSError =============================== warnings summary =============================== tests/sh_test.py::FunctionalTests::test_fg_alternative /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread STDIN thread for pid 2687 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2597, in input_thread done = stdin.write() File "/build/python-sh/src/sh/sh.py", line 2910, in write chunk = self.get_chunk() File "/build/python-sh/src/sh/sh.py", line 2851, in fn chunk = stdin.read(bufsize) File "/usr/lib/python3.14/site-packages/_pytest/capture.py", line 229, in read raise OSError( "pytest: reading from stdin while output is captured! Consider using `-s`." ) OSError: pytest: reading from stdin while output is captured! Consider using `-s`. Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py: 13 warnings /build/python-sh/src/sh/sh.py:1986: DeprecationWarning: This process (pid=2641) is multi-threaded, use of fork() may lead to deadlocks in the child. self.pid = os.fork() tests/sh_test.py::FunctionalTests::test_incremental_composition /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2694 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_120: RAN: /usr/bin/python /tmp/tmpbaaagec1 8 STDOUT: STDERR: Exception ignored while flushing sys.stdout: BrokenPipeError: [Errno 32] Broken pipe Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_multiple_pipes /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2720 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_1: RAN: /usr/bin/python -u /tmp/tmp7h513k7f STDOUT: STDERR: Traceback (most recent call last): File "/tmp/tmp7h513k7f", line 7, in sys.stdout.write(chr(ord(letter)+1)) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_multiple_pipes /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2725 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_1: RAN: /usr/bin/python -u /tmp/tmp7h513k7f STDOUT: STDERR: Traceback (most recent call last): File "/tmp/tmp7h513k7f", line 7, in sys.stdout.write(chr(ord(letter)+1)) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_done_callback /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2714 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_1: RAN: /usr/bin/python -u /tmp/tmp9kaue_zs STDOUT: STDERR: Traceback (most recent call last): File "/tmp/tmp9kaue_zs", line 7, in sys.stdout.write(l) ~~~~~~~~~~~~~~~~^^^ BrokenPipeError: [Errno 32] Broken pipe Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_unchecked_pipeline_failure /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2744 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_2: RAN: /usr/bin/python /tmp/tmp2yrya9bx STDOUT: STDERR: Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_piped_exception2 /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2761 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_2: RAN: /usr/bin/python /tmp/tmpqi2rwza5 STDOUT: STDERR: Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_piped_exception1 /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2784 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_2: RAN: /usr/bin/python /tmp/tmpn723s6r_ STDOUT: STDERR: Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_partially_applied_callback /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2902 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_1: RAN: /usr/bin/python -u /tmp/tmpmhq2hzod STDOUT: STDERR: Traceback (most recent call last): File "/tmp/tmpmhq2hzod", line 8, in print(letter) ~~~~~^^^^^^^^ BrokenPipeError: [Errno 32] Broken pipe Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_unchecked_producer_failure /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 2972 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_2: RAN: /usr/bin/python /tmp/tmpkp9v037f STDOUT: STDERR: Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::FunctionalTests::test_err_piped /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread background thread for pid 3053 Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1642, in wrap fn(*rgs, **kwargs) ~~^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2647, in background_thread handle_exit_code(exit_code) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 2338, in fn return self.command.handle_command_exit_code(exit_code) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 823, in handle_command_exit_code raise exc sh.ErrorReturnCode_1: RAN: /usr/bin/python -u /tmp/tmpifcyecyw STDOUT: STDERR: Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::MiscTests::test_threaded_with_contexts /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-19 (f1) Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/tests/sh_test.py", line 3451, in f1 results[0] = str(system_python("one")) ~~~~~~~~~~~~~^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1511, in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) File "/build/python-sh/src/sh/sh.py", line 717, in __init__ self.process = OProc( ~~~~~^ self, ^^^^^ ...<7 lines>... process_assign_lock, ^^^^^^^^^^^^^^^^^^^^ ) ^ File "/build/python-sh/src/sh/sh.py", line 2163, in __init__ raise ForkException(fork_exc) sh.ForkException: Original exception: =================== Traceback (most recent call last): File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ tty.setraw(self._stdout_child_fd) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/tty.py", line 61, in setraw mode = tcgetattr(fd) termios.error: (25, 'Inappropriate ioctl for device') Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) tests/sh_test.py::MiscTests::test_threaded_with_contexts /usr/lib/python3.14/site-packages/_pytest/threadexception.py:58: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-20 (f2) Traceback (most recent call last): File "/usr/lib/python3.14/threading.py", line 1082, in _bootstrap_inner self._context.run(self.run) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/usr/lib/python3.14/threading.py", line 1024, in run self._target(*self._args, **self._kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/build/python-sh/src/sh/tests/sh_test.py", line 3455, in f2 results[1] = str(system_python("two")) ~~~~~~~~~~~~~^^^^^^^ File "/build/python-sh/src/sh/sh.py", line 1511, in __call__ rc = self.__class__.RunningCommandCls(cmd, call_args, stdin, stdout, stderr) File "/build/python-sh/src/sh/sh.py", line 717, in __init__ self.process = OProc( ~~~~~^ self, ^^^^^ ...<7 lines>... process_assign_lock, ^^^^^^^^^^^^^^^^^^^^ ) ^ File "/build/python-sh/src/sh/sh.py", line 2163, in __init__ raise ForkException(fork_exc) sh.ForkException: Original exception: =================== Traceback (most recent call last): File "/build/python-sh/src/sh/sh.py", line 2045, in __init__ tty.setraw(self._stdout_child_fd) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.14/tty.py", line 61, in setraw mode = tcgetattr(fd) termios.error: (25, 'Inappropriate ioctl for device') Enable tracemalloc to get traceback where the object was allocated. See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) ../../../../usr/lib/python3.14/site-packages/_pytest/cacheprovider.py:475 /usr/lib/python3.14/site-packages/_pytest/cacheprovider.py:475: PytestCacheWarning: could not create cache path /build/python-sh/src/sh/.pytest_cache/v/cache/nodeids: [Errno 24] Too many open files: '/build/python-sh/src/sh/pytest-cache-files-3afoo7ku' ../../../../usr/lib/python3.14/site-packages/_pytest/cacheprovider.py:429 /usr/lib/python3.14/site-packages/_pytest/cacheprovider.py:429: PytestCacheWarning: could not create cache path /build/python-sh/src/sh/.pytest_cache/v/cache/lastfailed: [Errno 24] Too many open files: '/build/python-sh/src/sh/pytest-cache-files-ufeou39q' -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED tests/sh_test.py::FunctionalTests::test_custom_separator - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_stringio_output - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_err_redirection - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_with_context_nested - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_no_pipe - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_huge_piped_data - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_exit - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_bad_sig_raise_exception - sh.F... FAILED tests/sh_test.py::FunctionalTests::test_no_close_fds - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_timeout_wait_overstep - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_none_arg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_close_fds - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_new_session_new_group - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_no_err - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_incremental_composition - sh.F... FAILED tests/sh_test.py::FunctionalTests::test_non_ascii_error - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_ok_code - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_grandchild_no_sighup - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_no_wait - sh.F... FAILED tests/sh_test.py::FunctionalTests::test_short_option - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_custom_separator_space - sh.Fo... FAILED tests/sh_test.py::FunctionalTests::test_multiple_pipes - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_done_callback - sh.ForkExcepti... FAILED tests/sh_test.py::FunctionalTests::test_stdin_unbuffered_bufsize - sh.... FAILED tests/sh_test.py::FunctionalTests::test_ok_code_exception - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_async_exc - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_async_iter - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_doesnt_execute_directories - s... FAILED tests/sh_test.py::FunctionalTests::test_command_wrapper_equivalence - ... FAILED tests/sh_test.py::FunctionalTests::test_iter_generator - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_change_stdout_buffering - sh.F... FAILED tests/sh_test.py::FunctionalTests::test_custom_timeout_signal - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_cwd - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stdin_from_string - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_no_arg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_async_iter_exc - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_sigpipe - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_done_cb_exc - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_custom_long_prefix - sh.ForkEx... FAILED tests/sh_test.py::FunctionalTests::test_timeout_wait - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_terminate - sh... FAILED tests/sh_test.py::FunctionalTests::test_callable_stdin - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_exit_code - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_timeout - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_generator_and_callback - sh.Fo... FAILED tests/sh_test.py::FunctionalTests::test_patched_glob - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_timeout_overstep - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_pushd - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_quote_escaping - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_tty_input - OSError: [Errno 25... FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_line_unbuffered FAILED tests/sh_test.py::FunctionalTests::test_manual_stdin_iterable - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_ok_code_none - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_signal_group - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_timeout_wait_negative - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_binary_input - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_false_bool_ignore - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_async_return_cmd - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_async_return_cmd_exc - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_async - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_multiple_bakes - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_exit_code_with_hasattr - sh.Fo... FAILED tests/sh_test.py::FunctionalTests::test_callable_interact - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_err_to_out - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_iter_unicode - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_long_bool_option - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_pass_fds - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_piped_generator - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_partially_applied_callback - s... FAILED tests/sh_test.py::FunctionalTests::test_number_arg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_raw_args - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_output_equivalence - sh.ForkEx... FAILED tests/sh_test.py::FunctionalTests::test_fg_false - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stringio_input - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_internal_bufsize - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_arg_preprocessor - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_manual_stdin_queue - sh.ForkEx... FAILED tests/sh_test.py::FunctionalTests::test_ok_code_ignores_bad_sig_exception FAILED tests/sh_test.py::FunctionalTests::test_cwd_fg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_binary_pipe - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_err_redirection_actual_file - ... FAILED tests/sh_test.py::FunctionalTests::test_unicode_exception - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_signal_exception - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_failure_with_large_output - sh... FAILED tests/sh_test.py::FunctionalTests::test_arg_string_coercion - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_buffered - sh.... FAILED tests/sh_test.py::FunctionalTests::test_no_proc_no_attr - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_stdin_newline_bufsize - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_bg_exit_code - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_tty_output - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_manual_stdin_file - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_shadowed_subcommand - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_general_signal - sh.ForkExcept... FAILED tests/sh_test.py::FunctionalTests::test_multiple_args_long_option - sh... FAILED tests/sh_test.py::FunctionalTests::test_composition - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_exit_code_from_exception - sh.... FAILED tests/sh_test.py::FunctionalTests::test_empty_stdin_no_hang - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_cast_bg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_no_out_iter_err - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_background - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_with_input - s... FAILED tests/sh_test.py::FunctionalTests::test_command_wrapper - sh.ForkExcep... FAILED tests/sh_test.py::FunctionalTests::test_handle_both_out_and_err - sh.F... FAILED tests/sh_test.py::FunctionalTests::test_no_out - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_kill - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_short_bool_option - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_bake_args_come_first - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_with_context_args - sh.ForkExc... FAILED tests/sh_test.py::FunctionalTests::test_tty_stdin - OSError: [Errno 25... FAILED tests/sh_test.py::FunctionalTests::test_long_option - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_with_context - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_manual_stdin_string - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_background_exception - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_nonblocking_iter - sh.ForkExce... FAILED tests/sh_test.py::FunctionalTests::test_for_generator_to_err - sh.Fork... FAILED tests/sh_test.py::FunctionalTests::test_done_callback_no_deadlock - sh... FAILED tests/sh_test.py::FunctionalTests::test_subcommand_and_bake - sh.ForkE... FAILED tests/sh_test.py::FunctionalTests::test_decode_error_handling - sh.For... FAILED tests/sh_test.py::FunctionalTests::test_multiple_args_short_option - s... FAILED tests/sh_test.py::FunctionalTests::test_err_piped - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_trunc_exc - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_stdout_callback_line_buffered FAILED tests/sh_test.py::FunctionalTests::test_print_command - sh.ForkExcepti... FAILED tests/sh_test.py::FunctionalTests::test_unicode_arg - sh.ForkException: FAILED tests/sh_test.py::FunctionalTests::test_exception - sh.ForkException: FAILED tests/sh_test.py::MiscTests::test_percent_doesnt_fail_logging - sh.For... FAILED tests/sh_test.py::MiscTests::test_change_log_message - sh.ForkException: FAILED tests/sh_test.py::MiscTests::test_stdin_nohang - sh.ForkException: FAILED tests/sh_test.py::MiscTests::test_unicode_path - sh.ForkException: FAILED tests/sh_test.py::MiscTests::test_threaded_with_contexts - AssertionEr... FAILED tests/sh_test.py::MiscTests::test_fd_over_1024 - sh.ForkException: FAILED tests/sh_test.py::MiscTests::test_pickling - OSError: [Errno 24] Too m... FAILED tests/sh_test.py::MiscTests::test_eintr - OSError: [Errno 24] Too many... FAILED tests/sh_test.py::MiscTests::test_stop_iteration_doesnt_block - OSErro... FAILED tests/sh_test.py::ExecutionContextTests::test_no_interfere1 - OSError:... FAILED tests/sh_test.py::ExecutionContextTests::test_basic - OSError: [Errno ... FAILED tests/sh_test.py::ExecutionContextTests::test_multiline_defaults - OSE... FAILED tests/sh_test.py::ExecutionContextTests::test_set_in_parent_function FAILED tests/sh_test.py::ExecutionContextTests::test_no_interfere2 - OSError:... = 140 failed, 41 passed, 1 skipped, 1 deselected, 28 warnings in 167.54s (0:02:47) = ==> ERROR: A failure occurred in check(). Aborting... [!p]104[?7h]3008;end=8c2b248a2ffd412594cb3f9b2e026ef0\==> ERROR: Build failed, check /var/lib/archbuild/extra-riscv64/felix-0/build [?25h[?25h[?25hreceiving incremental file list python-sh-2.2.2-2-riscv64-build.log python-sh-2.2.2-2-riscv64-check.log sent 62 bytes received 59,348 bytes 39,606.67 bytes/sec total size is 2,373,097 speedup is 39.94