Using os.replace() leads to "Invalid cross-device link" on Core20
setups, due to having different mounts. Use shutil.copy() & os.remove()
instead.

Fixes the parse error for dict values that include ":"

COMMITS:
* Fix parsing of values with colons
* Update test_ovs.py
Fix pycodestyle pipeline
* Update ovs.py
One more pycodestyle error
* Update test_ovs.py
Add raw string to cure PEP8 fail
* tests:ovs: improve coverage for colon values

Co-authored-by: Pete MacKinnon <pmackinnon@nvidia.com>
Co-authored-by: Lukas Märdian <lukas.maerdian@canonical.com>

COMMITS:
* cli:apply: cleanup formatting
* cli:apply:utils: use networkctl reload/reconfigure
* utils: consolidate methods
* test:utils: improve coverage
* cli:utils: Review improvement for list handling
make use of list.extend() instead of individual list.append()s
Co-authored-by: Łukasz Zemczak <sil2100@vexillium.org>

This PR builds upon some functionality from #193, so that one should be merged first. It extends the existing CLI functionality of netplan set to auto-detect the filename of the corresponding YAML config which contains the original definition of this netdef. If, for example, there is a config file /etc/netplan/00-my-manual-config.yaml containing a definition for eth0 and we're calling netplan set network.ethernets.eth0.dhcp4=false, that setting will be overridden in 00-my-manual-config.yaml instead of masking it via 70-netplan-set.yaml.

If the set command contains multiple netdefs (e.g. netplan set network="{ethernets:{eth0:{dhcp4: true}}, bridges:{br0:{dhcp6: true}}}"), each will be checked individually and written to its corresponding file. If there is no prior definition of a given netdef, it will be written to 70-netplan-set.yaml by default.

Additionally, this PR introduces a small bug-fix to make netplan get network work, returning the full YAML tree, except for the top level network key.

In the C library libnetplan a new netplan_get_filename_by_id() API is introduced, parsing the current set of YAML configs and checking the resulting hashtable for the filename of the highest priority netdef.

fix match.original_name vs netdef->id shortcut (it shall be the same value, but not same reference/pointer)
Tracking of filename per netdef in the YAML parser
Improve wake-on-lan handling
Use NM aliases for wifi/ethernet keyfile groups
Refactoring NetworkManager keyfile generator to use GKeyFile instead of GString
Adding a NetworkManager keyfile parser
Adding a netplan YAML generator
New APIs introduced:

// YAML generator
void write_netplan_conf(const NetplanNetDefinition* def, const char* rootdir);
// NM keyfile parser
gboolean netplan_parse_keyfile(const char* filename, GError** error);
// general & utils
guint netplan_clear_netdefs();
gboolean netplan_delete_connection(const char* id, const char* rootdir);
gboolean netplan_generate(const char* rootdir);
gchar* netplan_get_id_from_nm_filename(const char* filename, const char* ssid);


== COMMITS: ==

* Prepare for NetworkManager YAML backend functionality (#181)

This is the first in a series of pull request to be prepared for setting up the functionality for a NetworkManager YAML/netplan backend, to be merged into slyon/networkmanager-yaml-backend before it will collectively be merged into master.

This PR refactors some of the YAML processing logic into libnetplan (from the generate binary), keeps track of the filename (absolute path) of the YAML file for each NetplanNetDefinition and implements a first additional library function netplan_get_id_from_nm_filename, which will output the NetplanNetDefinition ID given a path to a netplan generated .nmconnection keyfile in /run/NetworkManager/system-connections/.

The netplan ID is not stored inside NM's data structures, therefore we need a way to gain netplan's identifier for any given connection profile.

Commits:
* parse:generate: refactor process_yaml_hierarchy
* parse: track filename of netdefs
* parse: add netplan_get_id_from_filename library function
* parse: fix typo
* generate: some stylistic cleanup
* util: move netplan_get_id_from_nm_filename from parse.c

* Implement netplan_generate and netplan_delete_connection APIs (#182)

This is 2nd in a series of pull request implementing the functionality in libnetplan to provide a NetworkManager YAML backend. It builds upon #181

* It improves the netplan set CLI a bit, to delete a file, if only network: {version: 2} is left.
* It implements a netplan_generate function, which will call netplan generate in the background by spawning another process.
  * At some point the generate binary should be refactored, to call functions of libnetplan instead, so we could use this same (refactored) functionality inside the library directly
  * But for now this is as good as it gets...
* It implements a netplan_delete_connection function, which will delete a connection from the YAML structure of a file (or potentially the whole file, if it is empty afterwards), by utilizing the netplan set network.TYPE.IFNAME=NULL functionality by spawning another process.
  * At some point we should get rid of this Python/C split of the YAML parsers, to have all the functionality available inside the C library.
  * But for now this is as good as it gets, without duplicating any logic...

* NM: refactor keyfile generator, by using GLib's keyfile writer instead of custom writer (#184)

This is the 3rd in a series of pull requests to enable the implementation of a YAML/netplan backend for NetworkManager.

In this PR netplan's current keyfile generation code (in nm.c) is refactored to make use of GLib's keyfile writer, instead of using a custom, home grown approach. This way the keyfile settings can easily be read, written and overwritten (e.g. by the keyfile passthrough/fallback settings in #183) and it should be more robust overall.

While on it: Clean up the usage of keyfile alias for ethernet, wifi and wifi-security, which NetworkManager writes by default since a long time.

Commits:
* nm: refactoring to use GKeyFile writer, instead of home grown
* nm: uses alias for 'ethernet', 'wifi' and 'wifi-security'
NetworkManager writes the alias for 'ethernet' (802-3-ethernet), 'wifi' (802-11-wireless) and 'wifi-security' (802-11-wireless-security) settings by default since a long time, we should do so as well. Especially we should not mix and match.
see:
https://bugzilla.gnome.org/show_bug.cgi?id=696940 https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/c36200a225aefb2a3919618e75682646899b82c0


nm_keyfile_plugin_kf_set* from nm-keyfile-utils.c (libnm-core)

* Prepare passthrough mode for NM backend and YAML serializer (#187)

This is 4th in a series of pull requests. It enables parsing of the passthrough keyword inside the networkmanager backend settings, inside any netdef or wifi AP. Furthermore, it add a new serialize.c/h module, which can take a single netdef and render it into a YAML file. Testing is implemented by rendering a given YAML file into a new YAML file, and comparing the 1:1 match.

The YAML serializer does (by far) not yet support all supported netplan settings, but only the minimal set to enable the NetworkManager backend via passthrough mode.

The NM backend settings need to be available in every wifi AP definition (in addition to the netdef itself), as in the NetworkManager world, each wifi connection is a separate connection profile, while in the netplan world the wifi networks/AP are combined under a single device/netdef. But the settings passthrough (i.e. fallback mode) needs to be available for each individual NM connection profile, therefore we need them in the AP struct.

The passthrough setting is a mapping inside backend specific settings of key-value pairs, where the keys are separated by a dot in the format KEYFILE_GROUP.KEYFILE_KEY. Those values are passed through to the keyfile as is, and can be used as a fallback mechanism, where a specific feature is not yet implemented.

* Implement NM keyfile passthrough mode as a fallback mechanism (#183)

This is 5th in a series of pull requests to prepare for the NetworkManager YAML backend. It builds upon #187

It introduces a new (undocumented) top level type others, which needs to be used when a NM connection of unsupported type needs to be written. If used the connection.type needs to be specified by a passthrough setting, e.g.:

network:
  version: 2
  others:
    renderer: NetworkManager
    networkmanager:
      uuid: ...
      passthrough:
        connection.type: vxlan
        ipv4.method: auto
        ...
others is undocumented on purpose, as this is not supposed to be used in regular netplan configs, but can be used as a fallback mechanism, if a given network connection type is not yet implemented in netplan. It can then still be used via passthrough mode.
Furthermore, this PR introduced a new module nm-keyfile to libnetplan, which contains some logic to help with integration of the NetworkManager YAML settings backend. Especially the netplan_render_yaml_from_nm_keyfile function, which transforms a given GKeyFile* structure into a valid NetplanNetDefinition* structure and uses the serializer to transform this into a valid netplan YAML and save it to disk. For now this makes heavy use the the passthrough fallback mechanism, but it will be extended to place keyfile settings into the correct netplan schema in the future, step by step.

The nm.c module was extended to make use of the passthrough fallback mechanism, to render a valid NetworkManager keyfile out of this data and write it to disk. It uses the internal netplan data structure to generate the keyfile as usual, but allows to extend (or override) specified settings via passthrough if they are not (yet/fully) supported.

As discussed in #181, the netplan_get_id_from_nm_filename function is moved into this new nm-keyfile module from utils as this might be a better place for keyfile specific functionality.

Currently this PR is based upon the slyon/nm-4 branch. It will be rebased once PR #187 is merged, but the underlying base shouldn't change too much, so it should be fine to review this already.

Commits:
* nm-keyfile: Add netplan_render_yaml_from_nm_keyfile API
* configmanager: handle 'others' key
* nm: Implement fallback/passthrough mode
* nm-keyfile: support wifis, modems, bridges in addition to ethernets (and others)
* nm-keyfile: cleanup
* nm-keyfile: support bonds, vlans, tunnels
* validation: check others vs passthrough
* nm: fallback generator: allow dotted groups (i.e. wireguard-peer)
* nm: Update copyright
* nm-keyfile: handle connection-type alias
* WIP: nm: fallback integration
* nm-keyfile: modularize and clear handled fallback keys
* WIP: nm: fallback wifi_mode & interface_name
* WIP: netplan-yaml: draft YAML export
* NM: split into keyfile parser and YAML serializer
* nm-keyfile: cleanup
* nm: cleanup
* parse: cleanup
* nm: mention overriden settings in keyfile comment
* nm: allow to specify name (NM id) from backend_settings
* nm: simplify connection.type error handling
* test_nm_backend: adopt tests
* nm-keyfile: move netplan_get_id_from_nm_filename from util
* tests: move generator tests into generator/test_passthrough.py
* nm-keyfile: clear netdefs and improve docs
* nm: improve comments/docs
* nm-keyfile: support canonical names in addition to alias
* nm-keyfile: improve formatting
* doc: mention the  device type

* Improvements for the NetworkManager YAML backend integration (#189)

This PR is 6th in a series of pull requests to prepare for the NetworkManager YAML backend. It builds upon #183 and contains a few improvements and fixes, enabling the NetworkManager test-suite (i.e. make check) to fully pass, especially the tests in the patched keyfile plugin (src/settings/plugins/keyfile/tests/test-keyfile-settings).

It implements the following functionality:

Adding some basic support for the modems YAML schema, to allow matching of (physical) modem interfaces and enable the detection of GSM vs CDMA connections, which are distinct in NM while using the same definition in Netplan.
Switching the passthrough key-value pairs to a DataList (instead of HashTable), to keep order of the keyfile elements, which is relevant in some cases (e.g. for tc.qdiscs)
Avoid parsing and serialization of type other connections, as they might not contain the relevant handlers. Use full passthrough mode for those connections, to avoid parsing failures.
Allow independent modification of Netdef ID & match.name, to enable changing the connection.interface-name from nmcli, while keeping the netdef ID equal to a previously existing ID. This enables overriding of existing netdef ID by 90-NM-... YAML file, keeping the same Netdef ID.
Implement handling of keyfile UUIDs for each NM connection profile
Implement handling of empty keyfile groups (i.e. relevant for [bridge])

Commits:
* nm:nm-keyfile: allow to define empty Keyfile groups, like [bridge] or [proxy]
* passthrough: switch from GHashTable to GDataList, as some of the keyfile values need to be ordered, like tc.qdisks or tc.filters
* nm: special handling for [tc] group, where keys can contain dots
* serialize: write match stanza only for physical devices
* parse: modems are physical links and can have a 'match' stanza
* parse: improve clearing of netdefs
* nm: improve annotation of passthrouh settings
* nm-keyfile: improve removal of supported keys
Also, removes the group, if all of its keys were removed
* nm: improve detection of GSM connection
GSM/CDMA can also be defined on a type=bluetooth connection – not just
on MODEMS connections
* nm:nm-keyfile:serialize: set wakeonlan only for ethernet devices
* nm-keyfile:serialize: some modem stanzas
* nm-keyfile:serialize: handle OTHER type via full passthrough mode
* nm:nm-keyfile:serialize: improve interface-name/match handling
* nm: improve handling of NM uuid
* nm: improve passthrough.connection.type handling
* nm: improve handling of wifi.mode
* tests: add and adopt test cases
* nm-keyfile: allow to pass a previously known netdef ID, to enable overrides
* parse: allow changing match.name independenly of netdef ID
* nm-keyfile: update GSM vs CDMA comment
* parse: avoid creation of 'netdefs' hashmap in netplan_parse_yaml

* NM Integration: API update & Parser/Generator split (#191)

This is 7th in a series of pull requests to provide the API needed for implementation of a NetworkManager YAML backend. It builds upon #189

Changes contained in this PR:

*    Split the code/modules into a YAML/netplan generator (netplan.o in addition to nm.o, networkd.o, openvswitch.o, ...)
*    Split the code/modules into a keyfile/NM parser (parse-nm.o in addition to our YAML/netplan parser in parse.o)
*    Rename others top-level device type/stanza to nm-devices to make it more explicit and print a warning when using it
*    Move netplan_get_id_from_nm_filename back to utils.o, as it does not fit into the parse-nm.o module anymore (it is no parser specific functionality). Apply a small improvement to detect netplan_ids in testing scenario, where rootdir is provided.
*    Unify the new netplan generator module API with the other generators: void write_netplan_conf(const NetplanNetDefinition* def, const char* rootdir)
*    Unify the new keyfile parser module API with the other parser: gboolean netplan_parse_keyfile(const char* filename, GError** error)

* cleanup

Co-authored-by: Łukasz Zemczak <sil2100@vexillium.org>

Using arp-ip-targets as below is generating 10-netplan-bond0.netdev file which contains ARPIPTargets property with duplicated values;

arp-ip-targets:
- 10.10.10.10
- 20.20.20.20

ARPIPTargets=10.10.10.10 20.20.20.20 10.10.10.10 20.20.20.20

Duplicate values in ARPIPTargets prevents systemd-networkd to process the netdev file due to a validation error in Ubuntu 18.04LTS.

Commits:
* Prevent to have duplicate values set to ARPIPTargets in NetDev files (LP: #1915837)
* Added a test case to check ARPIPTargets value in NetDev files (LP: #1915837) (#199)
* Improved the test case to trigger multiple pass on the YAML config (LP: #1915837) (#199)
* Reduced the test case content to the required base minimum configuration items (LP: #1915837) (#199)
Co-authored-by: Kivanc Sunkar <kivanc@linxa.com>

Encapsulate wifi tests into own test class (requiring modprobe, i.e. isolation-machine level).
Make sure the management always stays up an connected (incl. DNS), independently of netplan

* tests: refactor WiFi parts into own class
* tests: keep management network up at all times during integration tests

Enable Github's CodeQL code scanning feature for periodic security checks.

* Enable CodeQL
* install build-dep to fix autobuild

Some protocols set the TTL field of the packet to 1; when passing through the tunnel, the packet is discarded. To solve the problem, the tunnel has the TTL option, but it was not in netplan.

According to https://bugs.launchpad.net/netplan/+bug/1846783

 this is required for IPIP/SIT/GRE tunnels.

Co-authored-by: kev1989 <krupenevev@gmail.com>

Add possibility to set initcwnd/initrwnd options on routes.

Commits:
* Added per route initcwnd/initrwnd option
* fixed integration test
* fixed integration test
* Update doc/netplan.md
Co-authored-by: Lukas Märdian <luk@slyon.de>
* Update doc/netplan.md
Co-authored-by: Lukas Märdian <luk@slyon.de>
* Update doc/netplan.md
Co-authored-by: Lukas Märdian <luk@slyon.de>
* Update src/parse.c
Co-authored-by: Lukas Märdian <luk@slyon.de>
* Update doc/netplan.md
Co-authored-by: Lukas Märdian <luk@slyon.de>
* Fixed formating and indentation
* Update tests/integration/routing.py
Co-authored-by: Lukas Märdian <luk@slyon.de>
* parse: rename congestion-window & advertised-receive-window
Co-authored-by: Marcin Matląg <mmatlag@sitel.net.pl>
Co-authored-by: Lukas Märdian <luk@slyon.de>
Co-authored-by: Lukas Märdian <lukas.maerdian@canonical.com>

In some cases the netifaces.ifaddresses() method returns an empty dict. Accessing this dict throws a KeyError, because there is no netifaces.AF_LINK (= 17) link layer address data.

Closes: #979266 (Debian)
systemd-resolve was renamed to resolvectl as of v239

Added information about the other accepted scalar duid which can be specified or is assumed if the option is omitted entirely.

They changed the term slave -> member.
https://github.com/openvswitch/ovs/commit/91fc374a9c5a2a4d9520aaa3588a7a18338a476e

Fixes LP: #1909114

The COMMON_BACKEND_HANDLERS have been forgotten for modem connections apparently. Add them to allow the definition of the special networkmanager: mapping, used for NetworkManager integration. We do not (yet) use that information (like uuid) in the current implementation. But reading YAML via NetworkManager will be broken if the networkmanager: mapping is not accepted.

* networkd: Fix changing of macaddress with systemd v247

* networkd: avoid writing MACAddress= [Link] into .link file (we already have it in .network file)

* network: some cleanup

Also update the Makefile to make clang accept the input (i.e. avoid passing header files to the compiler).

Commits:
* dbus: Remove unused variabes
This issue is seen when using clang to compile it
Same fix is needed for parse.c and networkd.c,
Fixes
src/dbus.c:49:23: error: unused variable 'stdout' [-Werror,-Wunused-variable]
g_autofree gchar *stdout = NULL;
^
Signed-off-by: Khem Raj <raj.khem@gmail.com>

* Makefile: Exclude .h files from target rule
This ensures that src/_features.h is not added to compiler cmdline which
can confuse the compilers e.g. clang as it may not like .h as valid input
and complain
| clang-11: error: cannot specify -o when generating multiple output files
| make: *** [Makefile:50: netplan-dbus] Error 1
| make: *** Waiting for unfinished jobs....
Signed-off-by: Khem Raj <raj.khem@gmail.com>

Renaming of network interfaces was not working when running netplan apply and interfaces would only be renamed after a reboot via udev. There is some udev-rename logic inside cli/commands/apply.py, which doesn't seem to be working, though.

This seems to be related to systemd-udevd's NamePolicy=keep default, which is set explicitly as of systemd v240 [0][1] and prevents renaming of interfaces via udev, if the name was set once (i.e. during boot). Even before v240 this has be the (implicit) default, so I'm not sure if this rename logic has ever worked...

I reworked/refactored the renaming of interfaces in apply.py to make use of iproute2 ip link set eth_OLD set name eth_NEW command instead and also added an integration test.

In order to rename interfaces via iproute2, they need to be DOWN, therefore the new code is bringing all to-be-renamed interfaces - which are not defined to be critical - down during apply and renames them accordingly.

Also, there was another issue with the detection of physical interfaces, where the system's current network interface names (from netifaces.interfaces()) have been compared to config_manager.py's physical_interfaces list, which are not (always) actual interface names (but netplan IDs), especially for netplan definitions with a match condition. So I added another util to lookup/match the actual interface name for a given match: condition.

[0] https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html
[1] https://github.com/systemd/systemd/blob/e62a7fea757f259eb330da5b6d3ab4ede46400a2/NEWS#L25

Commits:
* WIP: rename down interfaces on apply
* cli: improve rename of interfaces at runtime, during 'netplan apply'
* cli: fix linter
* WIP: refactor interface rename code
udevadm test-builtin net_setup_link seems to be only running once (at
boot time) and _not_ rename interfaces during 'netplan apply'
* cli: keep udev link changes like WakeOnLan=
* cli:utils: some cleanup
* cli: big cleanup + unit-tests
* tests: set-name on apply integration test
* cli: update comment about NamePolicy=keep rename

This is a fix for 1904633 netplan crashes while loading incorrect config.

Co-authored-by: Lukas Märdian <lukas.maerdian@canonical.com>

Add man pages for netplan get/set CLI and Config/Try/Cancel/Apply/Get/Set DBus methods.
This PR also adds a dbus-config feature flag, to indicate the availability of the DBus configuration management.

This implements the io.netplan.Netplan /io/netplan/Netplan io.netplan.Netplan Config method, which creates new /io/netplan/Netplan/config/<ID> objects, providing the following methods:

    io.netplan.Netplan.Config Get
    io.netplan.Netplan.Config Set
    io.netplan.Netplan.Config Try
    io.netplan.Netplan.Config Apply
    io.netplan.Netplan.Config Cancel
    And also the io.netplan.Netplan.Config Changed signal.

Each new config object will copy the current global netplan state (from /etc/netplan) to /tmp/netplan-config-<ID> then Get/Set can be called on that temp directory. On Apply/Try it will backup the current global config to /tmp/netplan-config-BACKUP and copy over the current config state to /etc/netplan. On revert the data from /tmp/netplan-config-BACKUP will be restored.

link-local handling needs to be properly implemented in the
NetworkManager backend.