backup / offline / archive a git repo

 I'm not sure why this isn't really covered simply in one place but here it goes...

Using just the cli you can do the following

a `git archive` command is not the thing we want, that is more for prepping code for deployment to a site…

more like https://docs.github.com/en/repositories/archiving-a-github-repository/backing-up-a-repository we want to offline or backup the repo

The idea is:

`git clone --mirror <repository_url>` which results in a dir `<repository_url>.git/`

`cd <repository_url>.git/`

`git bundle create ../<repository_name>.bundle --all` which makes a single file which can be cloned from later

compress it as you like `zstd --compress --progress --threads=0 --rm <repository_name>.bundle`

Then to get it all back you reverse the process

decompress the file `zstd --decompress --progress --threads=0 --rm <repository_name>.bundle.zst`

`git clone /path/to/<repository_name>.bundle /path/to/restored_repo`

and that should be it…

In practice, when I do this all the branches are remote still

```

  master

* newfeature

  remotes/origin/20190108-release

  remotes/origin/20220509-release

  remotes/origin/406-thing

```

but the remote is actually the bundle

```

ivan@srv:/ssd/git_repo_restored$ git remote -v

origin  /ssd/git_rep.bundle (fetch)

origin  /ssd/git_rep.bundle (push)

```

LXD/LXC networking a static crossover directly connected NIC

 For the life of me I couldn't find this out there on the net.  I looked at countless videos and read blogs but to no avail.

The task seemed easy, I have two physical hosts with dual-port 25GbE NICs.  I only have a 10GbE switch but want to take advantage of the faster link speed so why not use a DAC and just static IP the interfaces?!

Here is the setup:

srv1 & srv2 hosts are set up identically:

eth0_10gbe (NIC) --> br0_10gbe (host bridge) 10.10.0.0/24

eth1_25gbe (NIC) --> br1_25gbe (host bridge) 10.25.0.0/24

I then setup LXD to use the 10GbE bridge for networking using the unmanaged (by lxd) host bridge by using the lxd bridge nic.  This allows the DHCP request to be passed to my LAN DHCP server (Ubiquity UDMPro) for that particular segment (vlan tagged network).  This was relatively easy and there are a bunch of tutorials and things since this is rather standard.  

In Ubuntu I used netplan to add the VLANs to the host bridge and I use the subsequent "sub interface" in the container.  Said differently, UDMPro sends untagged traffic to the port, the NIC puts that in the host bridge which then breaks out the VLANs, then LXC containers are attached to the VLAN interfaces.

The NIC direct connected between hosts was super simple to configure for the host.  I was scping and stuff at 25GbE in mins.  Then trying to get LXC containers to do the same thing proved a little more challenging.

I eventually settled on creating a bridge (as stated above) and adding that to the container as eth1 using LXC's bridging nictype:

lxc config device add <container_name> eth1 nic nictype=bridged parent=br1_25gbe

I then need to go into the container OS and manually configure an IP.

Once of the LXD folks wrote a blog about some of the other kind of networking and includes code for doing manual static IP addressing from an LXC profile which is likely what I'll do next... all of my LXC containers are provisioned via terraform and then configured via ansible so maybe I'll go a different route (pun intended) but not sure yet.

But why though? This is all to allow for some of the containerized services to use the 25GbE network as a backhaul connection for their cluster traffic.

LXD (now Canonical's) a journey into the simplistically complex

The history of LXD is not part of this post...

LXC (Linux Containers) were not the goal.  I have experience with OpenStack and wanted to go that route but the hardware requirements and overhead for my particular project were so small (just a few different services) that someone on my team said "I'm just gonna install them all on my mac and be done with it.

I couldn't let that go, I needed some CI/CD and config management, I needed repeatable code based infrastructure!

I first found Canonical's MicroCloud and was kinda sold!  But their hardware requirement was min of three like servers and we just had two like servers.  So I did flirt with setting up LXD then putting a multi node MicroCloud cluster in it on VMs but that seemed a little crazy.

So I started with one host, configured LXD and started building containers with terraform and ansible.  It worked so well I was convinced that this was the right path forward.

The hardware are some Ryzen 5 boxes with max RAM and a few SSDs and a dual port 25GbE NIC.  One port from the NICs are direct attached to each other (no switching) and the other is into a 10GbE switch since that is what we have.

I configured each LXC server as a remote of the other and added the remote using the 25GbE network so copy/move operations would be max speed.

I had some trouble getting that part working since each time I would follow the steps to add the remote using the trust cert I would get an error.  No amount of googling found the answer so I tried to post on the LXD forums at Canonical... that was a waste of time since I couldn't even post and there was a disclaimer about "not for tech support" bla bla... so Reddit for the win!

I created a post "LXD to LXD host on one NIC, everything else on another?" in the LXD subreddit and within mins I was off and going!

I can run lxc commands from my laptop connected wifi and see both remotes and issue commands, it's wicked simple once you get it all setup and memorize all the CLI commands ;) 

I opted to have the 10GbE connected to the switch getting all the untagged VLAN traffic then use a linux bridge to pass the VLANs as networks into the containers.  I kinda followed the OpenStack pattern from the old days but also found Trevor Sullivan's YouTube Tutorials super helpful as well as the OG PM of LXD Stéphane Graber's YouTube and Site very helpful!

I might add more details in the future, but my hope is if anyone else is looking for some LXD/LXC servers that are not clustered and hit some of these road blocks maybe this will shorten their search a little.

MySQL dumps from cloudSQL, transform to csv, then load infile locally

I got some databases I need to migrate from one server to another.  We opted for the good old fashion `mysqldump`

So far it is wicked slow to import one of the tables, while working through the solution I needed to transform the data from the dump format of insert statements per line to a csv file.  Yeah, I could have just done the export again into this new format but why bother with all that and the network egress fees again, lets just transform what we have.

I asked ChatGPT to come up with some script to do this for me... that script failed :(

I ended up basically just settling on the good old fashion grep & sed commands:

time grep "INSERT INTO" /path/to/file_src.sql | sed -e 's/^[^(]*[(]//' -e 's/),(/\n/g' -e 's/);$//' > /path/to/file_dest.csv

so with this one command you consider only the lines with INSERT commands and you remove the front part something like: "INSERT INTO `table` ("

You also "split" and make new lines on "),(" which is the "boundary" between all the rows

Then we trim the end ");"

So in this one-liner we do basically all we need... if you have hundreds of gigs of tables like me it's faster to run this than re-export it all over the net.

We then needed to import it so I manually took the create table part of the dump, added it into another file and wrapped it in a loop.  I experimented with additional MySQL tuning bits and bobs over the iterations and have no idea if it is helping or not.

Find more and the scripts in this gist https://gist.github.com/ivanlawrence/c3fdbdcab0a34df714f0361f5f55e721

Google Workspace Routing: gmail pro catch-all routing

 I like to do this dumb thing where I use unique email addresses per service.  It started to thwart organizations from selling my email address, but now everything has been breached it has a unique benefit that I don't have the same username at every site so searching through my password manager is fast and also having someone try to login as `microsoft@example.com` but on google has nothing to do with my real account.

Well, I've moved my DNS registrar and name servers around a bit now that Google Domains has been sold.  Up shot is CloudFlare Domain Name Registration is a little cheaper and their API is very easy to deal with if you wanna do a roll your own Dynamic DNS service or something.

When I moved DNS some of my emails started to bounce :(

Looks like there was a gmail default route for "Google Domains Email Forwarding" but I couldn't find anything about it in the docs Set up Default routing for your organization which I think I was using before but now was no longer able to route since I wasn't using Google Domains?

The changes I made were for All Recipients > Envelope recipient > Change envelope recipient > Recipient username > my_username

When tested this seemed to give the result I wanted where the previously bounced emails like `foo@example.com` now was getting sent to `my_username@example.com`.

I thought I would share it here in case I forget and need to do it again.  I have other default routes for other domains and one of them adds a recipient instead of changing the recipient and that kinda dirties the email headers (more than they are now) but it's also a way.

MixPanel: use mixpanel.people.set() and mixpanel.identify() together

Found at https://community.mixpanel.com/questions/585/how-to-implement-peopleset-in-js-api.html

Question

stephendb9 asked · Jan 22 at 11:45 AM

How to implement people.set() in JS API

What is the right way to implement tracking for People properties in JS? The documentation states:

In order to send profile updates, you must call mixpanel.identify. The Mixpanel library does not automatically create people profiles for any user that performs an event; you have to explicitly call mixpanel.identify, which empowers you to only create profiles for registered users.

But in the examples, identify() is called before people.set() and sometimes it's after. Or not at all.

https://developer.mixpanel.com/docs/javascript#section-setting-profile-properties

• What is correct sequence of calls?
• Does identify() have to be called every time you want to change a property? Or just once per Profile?
• If identify() was already called for this Profile, do you just call it again with the same id?
• What if this Profile was created with alias()?

apipeople profilespeople properties

 1

1 Answer

annecamacho answered · Mar 11 at 2:27 PM

@stephendb9 -

The mixpanel.identify() call when paired with a mixpanel.people.set() call is intended to flush the data to Mixpanel, and it is required anytime that you intend to create or update a Mixpanel people property.

What is correct sequence of calls?

• You just need to call identify in the same function/context that you call people.set — the actual sequence in your code is less critical.

Does identify() have to be called every time you want to change a property? Or just once per Profile?

• Identify needs to be called every time you make a people.set call — to either create or update a people profile within Mixpanel.

If identify() was already called for this Profile, do you just call it again with the same id?

• If you've already identified the user you do not need to pass anything through the identify function, as you are just using it to flush the data. My personal preference is to always have an empty identify call i.e. mixpanel.identify() vs mixpanel.identify(xyz) after every people.set to ensure the data is sent to Mixpanel and stored to the profile.

What if this Profile was created with alias()?

• If you are aliasing or identifying a user, then a profile will not be created, unless you are also doing a people.set. Alias and identify methods are intended to help manage the identities of your users throughout their interactions in your site or app, but they also serve a second purpose of flushing the data to Mixpanel. The alias method, similar to identify, has a flush mechanism, but should only be used for the initial people.set calls when users are registering or signing up for an account. Whenever you add/update future people properties with people.set, you would call identify() to flush the data.

For more details on the alias and identify methods, here is a great overview.

-Anne

gcloud ADC for local testing (and how I will never get that day back)

After spending a day fighting permissions errors while running some test code locally using an account that I know for sure has the correct permissions:
{ Error: 7 PERMISSION_DENIED: User not authorized to perform this action.
at Object.exports.createStatusError (/Users/ivan/app/node_modules/grpc/src/common.js:87:15)
at Object.onReceiveStatus (/Users/ivan/app/node_modules/grpc/src/client_interceptors.js:1188:28)
at InterceptingListener._callNext (/Users/ivan/app/node_modules/grpc/src/client_interceptors.js:564:42)
at InterceptingListener.onReceiveStatus (/Users/ivan/app/node_modules/grpc/src/client_interceptors.js:614:8)
at callback (/Users/ivan/app/node_modules/grpc/src/client_interceptors.js:841:24)
code: 7,
metadata: Metadata { _internal_repr: {} },
details: 'User not authorized to perform this action.',
note: 'Exception occurred in retry method that was not classified as transient' }

I was even able to use gcloud to post a message to the same topic my code was trying to use just doing it from the CLI!  no permission errors there!

So after rebuilding and reconfiguring my environment, not to mention reinstalling gcloud and changing the configs a ton of times... I found out that 'gcloud auth login' no longer creates ADC for you... you must use
“gcloud auth application-default login”

This takes you through a browser based auth for the 'Google Auth Library' and BLAM-O you are in business!