Can't assign ip address to veth directly after creating veth pair












0















If I run the following, then the ip address doesn't actually get added:



ip link add veth0 type veth peer name veth1; ip address add 10.0.1.2/24 dev veth1


However if I sleep in between the commands, then it does get added:



ip link add veth0 type veth peer name veth1; sleep 0.1; ip address add 10.0.1.2/24 dev veth1


I've repeated multiple times and the results are consistent.



Is there a nicer way of doing this rather than just sleeping in between?



UPDATE:



Using ip montitor, it seems that a load of stuff is being done when the veth pair is created (no idea what), and that when both commands are run on the same line (without sleep), the ip address creation stops a lot of this stuff being done and the ip addresses get automatically deleted.



Does anyone understand what all this stuff is which happens when the veth pair is created?



output of ip monitor for veth creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
372: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
373: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
372: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
373: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium


ip monitor output for adding ip address:



372: veth1    inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2


ip monitor output when running both on same line (without sleep). The *'s indicate the line which is the start of the ip address creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
374: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
375: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
****************************************************************
374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
****************************************************************
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
374: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
ff00::/8 dev veth1 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted 10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
Deleted 374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
Deleted broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link
valid_lft forever preferred_lft forever
local fe80::1ab9:60e6:dfe2:5904 dev veth1 table local proto kernel metric 0 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 102 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link
valid_lft forever preferred_lft forever
local fe80::aa07:cb3f:757e:f3f4 dev veth0 table local proto kernel metric 0 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 101 pref medium









share|improve this question

























  • Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

    – grawity
    Jan 12 at 9:52











  • It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

    – dirkt
    Jan 12 at 12:10











  • @grawity Thanks for that, I've updated the question with the ip monitor output

    – Jonny Shanahan
    Jan 12 at 12:16
















0















If I run the following, then the ip address doesn't actually get added:



ip link add veth0 type veth peer name veth1; ip address add 10.0.1.2/24 dev veth1


However if I sleep in between the commands, then it does get added:



ip link add veth0 type veth peer name veth1; sleep 0.1; ip address add 10.0.1.2/24 dev veth1


I've repeated multiple times and the results are consistent.



Is there a nicer way of doing this rather than just sleeping in between?



UPDATE:



Using ip montitor, it seems that a load of stuff is being done when the veth pair is created (no idea what), and that when both commands are run on the same line (without sleep), the ip address creation stops a lot of this stuff being done and the ip addresses get automatically deleted.



Does anyone understand what all this stuff is which happens when the veth pair is created?



output of ip monitor for veth creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
372: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
373: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
372: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
373: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium


ip monitor output for adding ip address:



372: veth1    inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2


ip monitor output when running both on same line (without sleep). The *'s indicate the line which is the start of the ip address creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
374: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
375: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
****************************************************************
374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
****************************************************************
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
374: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
ff00::/8 dev veth1 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted 10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
Deleted 374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
Deleted broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link
valid_lft forever preferred_lft forever
local fe80::1ab9:60e6:dfe2:5904 dev veth1 table local proto kernel metric 0 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 102 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link
valid_lft forever preferred_lft forever
local fe80::aa07:cb3f:757e:f3f4 dev veth0 table local proto kernel metric 0 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 101 pref medium









share|improve this question

























  • Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

    – grawity
    Jan 12 at 9:52











  • It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

    – dirkt
    Jan 12 at 12:10











  • @grawity Thanks for that, I've updated the question with the ip monitor output

    – Jonny Shanahan
    Jan 12 at 12:16














0












0








0








If I run the following, then the ip address doesn't actually get added:



ip link add veth0 type veth peer name veth1; ip address add 10.0.1.2/24 dev veth1


However if I sleep in between the commands, then it does get added:



ip link add veth0 type veth peer name veth1; sleep 0.1; ip address add 10.0.1.2/24 dev veth1


I've repeated multiple times and the results are consistent.



Is there a nicer way of doing this rather than just sleeping in between?



UPDATE:



Using ip montitor, it seems that a load of stuff is being done when the veth pair is created (no idea what), and that when both commands are run on the same line (without sleep), the ip address creation stops a lot of this stuff being done and the ip addresses get automatically deleted.



Does anyone understand what all this stuff is which happens when the veth pair is created?



output of ip monitor for veth creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
372: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
373: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
372: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
373: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium


ip monitor output for adding ip address:



372: veth1    inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2


ip monitor output when running both on same line (without sleep). The *'s indicate the line which is the start of the ip address creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
374: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
375: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
****************************************************************
374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
****************************************************************
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
374: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
ff00::/8 dev veth1 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted 10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
Deleted 374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
Deleted broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link
valid_lft forever preferred_lft forever
local fe80::1ab9:60e6:dfe2:5904 dev veth1 table local proto kernel metric 0 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 102 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link
valid_lft forever preferred_lft forever
local fe80::aa07:cb3f:757e:f3f4 dev veth0 table local proto kernel metric 0 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 101 pref medium









share|improve this question
















If I run the following, then the ip address doesn't actually get added:



ip link add veth0 type veth peer name veth1; ip address add 10.0.1.2/24 dev veth1


However if I sleep in between the commands, then it does get added:



ip link add veth0 type veth peer name veth1; sleep 0.1; ip address add 10.0.1.2/24 dev veth1


I've repeated multiple times and the results are consistent.



Is there a nicer way of doing this rather than just sleeping in between?



UPDATE:



Using ip montitor, it seems that a load of stuff is being done when the veth pair is created (no idea what), and that when both commands are run on the same line (without sleep), the ip address creation stops a lot of this stuff being done and the ip addresses get automatically deleted.



Does anyone understand what all this stuff is which happens when the veth pair is created?



output of ip monitor for veth creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
372: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
373: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
372: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
373: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
372: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether da:a6:13:1a:29:31 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
373: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 4a:9e:6c:13:13:34 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium


ip monitor output for adding ip address:



372: veth1    inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2


ip monitor output when running both on same line (without sleep). The *'s indicate the line which is the start of the ip address creation:



ipv4 dev veth1 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
ipv6 dev veth1 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
374: veth1@NONE: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ipv4 dev veth0 forwarding on rp_filter strict mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
ipv6 dev veth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off
375: veth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
****************************************************************
374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
****************************************************************
local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
374: veth1@veth0: <NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
ff00::/8 dev veth1 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth0 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted 10.0.1.0/24 dev veth1 proto kernel scope link src 10.0.1.2
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
Deleted 374: veth1 inet 10.0.1.2/24 scope global veth1
valid_lft forever preferred_lft forever
Deleted broadcast 10.0.1.255 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted broadcast 10.0.1.0 dev veth1 table local proto kernel scope link src 10.0.1.2
Deleted local 10.0.1.2 dev veth1 table local proto kernel scope host src 10.0.1.2
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
ff00::/8 dev veth1 table local metric 256 pref medium
375: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether b6:7f:f2:e5:69:e6 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
374: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 72:94:ec:5a:d7:76 brd ff:ff:ff:ff:ff:ff
Deleted ff00::/8 dev veth1 table local metric 256 pref medium
ff00::/8 dev veth1 table local metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link
valid_lft forever preferred_lft forever
local fe80::1ab9:60e6:dfe2:5904 dev veth1 table local proto kernel metric 0 pref medium
374: veth1 inet6 fe80::1ab9:60e6:dfe2:5904/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 256 pref medium
fe80::/64 dev veth1 proto kernel metric 102 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link
valid_lft forever preferred_lft forever
local fe80::aa07:cb3f:757e:f3f4 dev veth0 table local proto kernel metric 0 pref medium
375: veth0 inet6 fe80::aa07:cb3f:757e:f3f4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Deleted fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 101 pref medium






linux networking ip-address network-interface virtual-network






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 12 at 12:16







Jonny Shanahan

















asked Jan 12 at 9:44









Jonny ShanahanJonny Shanahan

11




11













  • Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

    – grawity
    Jan 12 at 9:52











  • It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

    – dirkt
    Jan 12 at 12:10











  • @grawity Thanks for that, I've updated the question with the ip monitor output

    – Jonny Shanahan
    Jan 12 at 12:16



















  • Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

    – grawity
    Jan 12 at 9:52











  • It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

    – dirkt
    Jan 12 at 12:10











  • @grawity Thanks for that, I've updated the question with the ip monitor output

    – Jonny Shanahan
    Jan 12 at 12:16

















Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

– grawity
Jan 12 at 9:52





Can you retry the first command, but while running ip monitor in another terminal? Does it show the address being added and quickly removed, or does it show nothing at all?

– grawity
Jan 12 at 9:52













It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

– dirkt
Jan 12 at 12:10





It's a race condition: The kernel must finish adding the card before it can configure it, even if the ip command returns. If for some reason adding is slow, you must wait until it appears. You can loop on ip link show | grep veth1 etc., but if sleep already works...

– dirkt
Jan 12 at 12:10













@grawity Thanks for that, I've updated the question with the ip monitor output

– Jonny Shanahan
Jan 12 at 12:16





@grawity Thanks for that, I've updated the question with the ip monitor output

– Jonny Shanahan
Jan 12 at 12:16










0






active

oldest

votes











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393461%2fcant-assign-ip-address-to-veth-directly-after-creating-veth-pair%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Super User!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393461%2fcant-assign-ip-address-to-veth-directly-after-creating-veth-pair%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

How to make a Squid Proxy server?

Is this a new Fibonacci Identity?

19世紀