Explore all Firewall open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in Firewall

opensnitch

v1.5.0

fail2ban

0.11.2 (2020/11/23) - heal-the-world-with-security-tools

ModSecurity

v2.9.5

naxsi

1.3

simplewall

Popular Libraries in Firewall

opensnitch

by evilsocket doticonpythondoticon

star image 7096 doticonGPL-3.0

OpenSnitch is a GNU/Linux port of the Little Snitch application firewall

fail2ban

by fail2ban doticonpythondoticon

star image 6361 doticonGPL-2.0

Daemon to ban hosts that cause multiple authentication errors

TheFatRat

by screetsec doticoncdoticon

star image 5619 doticonGPL-3.0

Thefatrat a massive exploiting tool : Easy tool to generate backdoor and easy tool to post exploitation attack like browser attack and etc . This tool compiles a malware with popular payload and then the compiled malware can be execute on windows, android, mac . The malware that created with this tool also have an ability to bypass most AV software protection .

TheFatRat

by Screetsec doticoncdoticon

star image 5320 doticonGPL-3.0

Thefatrat a massive exploiting tool : Easy tool to generate backdoor and easy tool to post exploitation attack like browser attack and etc . This tool compiles a malware with popular payload and then the compiled malware can be execute on windows, android, mac . The malware that created with this tool also have an ability to bypass most AV software protection .

ModSecurity

by SpiderLabs doticonc++doticon

star image 5316 doticonApache-2.0

ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx that is developed by Trustwave's SpiderLabs. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis. With over 10,000 deployments world-wide, ModSecurity is the most widely deployed WAF in existence.

Awesome-WAF

by 0xInfection doticonpythondoticon

star image 4088 doticonApache-2.0

๐Ÿ”ฅ Everything about web-application firewalls (WAF).

naxsi

by nbs-system doticoncdoticon

star image 3789 doticonGPL-3.0

NAXSI is an open-source, high performance, low rules maintenance WAF for NGINX

esp_wifi_repeater

by martin-ger doticoncdoticon

star image 3677 doticonMIT

A full functional WiFi Repeater (correctly: a WiFi NAT Router)

pfsense

by pfsense doticonphpdoticon

star image 3415 doticonApache-2.0

Main repository for pfSense

Trending New libraries in Firewall

ngx_waf

by ADD-SP doticoncdoticon

star image 918 doticonBSD-3-Clause

Handy, High performance, ModSecurity compatible Nginx firewall module & ๆ–นไพฟใ€้ซ˜ๆ€ง่ƒฝใ€ๅ…ผๅฎน ModSecurity ็š„ Nginx ้˜ฒ็ซๅข™ๆจกๅ—

gsocket

by hackerschoice doticoncdoticon

star image 625 doticonBSD-2-Clause

Connect like there is no firewall. Securely.

ebpfsnitch

by harporoeder doticonc++doticon

star image 588 doticonBSD-3-Clause

Linux Application Level Firewall based on eBPF and NFQUEUE.

Xeexe-TopAntivirusEvasion

by persianhydra doticonpythondoticon

star image 463 doticonGPL-3.0

Undetectable & Xor encrypting with custom KEY (FUD Metasploit Rat) bypass Top Antivirus like BitDefender,Malwarebytes,Avast,ESET-NOD32,AVG,... & Automatically Add ICON and MANIFEST to excitable

coraza-waf

by jptosso doticongodoticon

star image 345 doticonApache-2.0

Coraza WAF is a golang modsecurity compatible web application firewall library

nftables-nat-rust

by arloor doticonrustdoticon

star image 313 doticonMIT

nftables nat rule generatorโ€”โ€”nftables nat่ง„ๅˆ™็”Ÿๆˆๅ™จ

ScareCrow-CobaltStrike

by GeorgePatsias doticonpythondoticon

star image 277 doticonMIT

Cobalt Strike script for ScareCrow payloads (EDR/AV evasion)

BlueGate

by ollypwn doticonpythondoticon

star image 243 doticon

PoC (DoS + scanner) for CVE-2020-0609 & CVE-2020-0610 - RD Gateway RCE

udm-kernel-tools

by fabianishere doticonshelldoticon

star image 226 doticonGPL-2.0

Tools for bootstrapping custom kernels on the UniFi Dream Machine

Top Authors in Firewall

1

PaloAltoNetworks

9 Libraries

star icon492

2

joyent

4 Libraries

star icon20

3

CoolerVoid

4 Libraries

star icon489

4

asalih

3 Libraries

star icon179

5

Kr328

3 Libraries

star icon122

6

douaneapp

3 Libraries

star icon19

7

subgraph

3 Libraries

star icon126

8

farcompen

3 Libraries

star icon15

9

gamemann

3 Libraries

star icon46

10

CyCoreSystems

3 Libraries

star icon99

1

9 Libraries

star icon492

2

4 Libraries

star icon20

3

4 Libraries

star icon489

4

3 Libraries

star icon179

5

3 Libraries

star icon122

6

3 Libraries

star icon19

7

3 Libraries

star icon126

8

3 Libraries

star icon15

9

3 Libraries

star icon46

10

3 Libraries

star icon99

Trending Kits in Firewall

No Trending Kits are available at this moment for Firewall

Trending Discussions on Firewall

Android Studio BumbleBee pair wifi not working

TLS v1.2 Cipher Suites in .NET 6 / GET Request Timeout

Private GKE cluster behind firewall getting calls from external IP

Is there a new solution for downloading package and dependencies for a given R version

Kubernetes NodePort is not available on all nodes - Oracle Cloud Infrastructure (OCI)

How can we mount a file as read-only in Linux through Go?

Terraform: How to modify a public subnet's route table that was created by module 'vpc'?

Why is ArgoCD confusing GitHub.com with my own public IP?

New-CimSession without elevation by providing admin credentials on Windows 10?

Failing to run Tomcat in IntelliJ IDEA

QUESTION

Android Studio BumbleBee pair wifi not working

Asked 2022-Apr-03 at 10:29

I am trying to connect my Android 11 device with android studio over adb wifi but it is not working.

I updated to latest stable bumblebee and updated my SDK I tried turning off firewall on my pc but it is same result.

When I use QR code method my android phone just shows "pairing device" and nothing happens If I try the code method, android studio just shows "searching for devices" but nothing happens

and, yes, I enabled wireless debugging on my phone and I am connected to the same wifi network.

I don't know if the problem is with my computer or phone. I do not have any other Android11+ phone to try with

ANSWER

Answered 2022-Jan-30 at 21:44

I was having the same problem as you. Neither pairing by QR nor by pairing code worked.

So I tried connecting by typing adb connect [phone_ip]:[port] in the terminal and that worked flawlessly. Didn't even need to plug the phone to the computer with USB. Your phone will tell you the IP and port right above the "pair with QR code" option inside the Wifi-debugging setting. Just connect to that address.

Source https://stackoverflow.com/questions/70905560

QUESTION

TLS v1.2 Cipher Suites in .NET 6 / GET Request Timeout

Asked 2022-Mar-30 at 12:52

I am currently trying to connect to an AWS REST API which requires at least TLS v1.2. The documentation stats that clients must also support cipher suites with perfect forward secrecy (PFS) such as Ephemeral Diffie-Hellman (DHE) or Elliptic Curve Ephemeral Diffie-Hellman (ECDHE).

When sending a GET request using the HttpClient, the connection simply times out. I have set the TLS version explicitely to TLSv1.2 like this:

1httpClientHandler.SslProtocols = SslProtocols.Tls12;
2

This works, I can see in the Wireshark trace that the correct TLS version is used. I have also confirmed that there is no firewall issue or similar.

Working Example (CURL)

When using cURL, I can see that the cipher suite in the Sever Hello response is TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030), which is also what the server requires.

enter image description here

Broken Example (.NET 6 with HttpClient)

When using the HttpClient in .NET 6, the above mentioned cipher suite is offered in the Client Hello, but the server response uses all of a sudden TLS_RSA_WITH_AES_256_GCM_SHA384:

enter image description here

I can see that there are additional extensions in the cURL request, for example Extension: psk_key_exchange_modes. Are there any explanations for why the server does not except the first cipher suite? From my understanding, the first offered cipher suite should be the preferred one, is that correct?

Is there a way to force a certain cipher suite in .NET 6?

This is the example I use to reproduce the issue:

1httpClientHandler.SslProtocols = SslProtocols.Tls12;
2public async void PollUrl(string url)
3{
4    HttpResponseMessage msg = new HttpResponseMessage();
5
6    ServicePointManager.Expect100Continue = true;
7    ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;
8
9    using HttpClientHandler httpClientHandler = new();
10
11    httpClientHandler.ServerCertificateCustomValidationCallback = (message, cert, chain, sslPolicyErrors) => true;
12    httpClientHandler.SslProtocols = SslProtocols.Tls12;
13
14    using HttpClient client = new(httpClientHandler);
15
16    // This content type is required for the API call
17    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
18
19    try
20    {
21        client.Timeout = TimeSpan.FromSeconds(5);
22        msg = await client.GetAsync(url);
23    }
24    catch (Exception e)
25    {
26        Console.WriteLine(e);
27    }
28
29    string stringValue = await msg.Content.ReadAsStringAsync();
30    Console.WriteLine(stringValue);
31}
32

The application is running on Server 2016.

ANSWER

Answered 2022-Mar-30 at 12:52

We finally found the reason for this. Windows did not have the required cypher suites enabled. We have used IISCrypto to enable the corresponding cypher suites and all is ok now.

It looks like it's possible to force .NET to TLS 1.2, even though it was not enabled on the server itself.

Source https://stackoverflow.com/questions/70338951

QUESTION

Private GKE cluster behind firewall getting calls from external IP

Asked 2022-Mar-24 at 20:28

We are getting logs that calls to k8s are being made, despite our cluster being private, as well as being behind the gcp firewall with a rule that blocks all ingress except IAP IPs (and ICMP). What am I missing?

1"protoPayload":{
2   "@type":"type.googleapis.com/google.cloud.audit.AuditLog"
3   "authenticationInfo":{
4      "principalEmail":"system:anonymous"
5   }
6   "authorizationInfo":["0":{2}]
7   "methodName":"io.k8s.post"
8   "requestMetadata":{
9      "callerIp":"45.*.*.*"
10      "callerSuppliedUserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
11   }
12   "resourceName":"Autodiscover/Autodiscover.xml"
13   "serviceName":"k8s.io"
14   "status":{
15      "code":"7"
16      "message":"Forbidden"
17   }
18}
19

ANSWER

Answered 2022-Mar-24 at 20:28

The private clusters have a control plane private endpoint and a control plane public endpoint and you can choose to disable the control plane public endpoint, this is the highest level of restricted access. So you can manage the cluster with the private endpoint internal IP address with tools like kubectl and any VM that uses the same subnet that your cluster can also access the private endpoint.However, it is important to say that even if you disable the public endpoint access, Google can use the control plane public endpoint for cluster management purposes, such as scheduled maintenance and automatic control plane upgrades. If you need more information about how to create a private cluster with public endpoint disable, you can consult the following public document.

You can review your public endpoints with the following command:

1"protoPayload":{
2   "@type":"type.googleapis.com/google.cloud.audit.AuditLog"
3   "authenticationInfo":{
4      "principalEmail":"system:anonymous"
5   }
6   "authorizationInfo":["0":{2}]
7   "methodName":"io.k8s.post"
8   "requestMetadata":{
9      "callerIp":"45.*.*.*"
10      "callerSuppliedUserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
11   }
12   "resourceName":"Autodiscover/Autodiscover.xml"
13   "serviceName":"k8s.io"
14   "status":{
15      "code":"7"
16      "message":"Forbidden"
17   }
18}
19gcloud container clusters describe YOUR_CLUSTER_NAME
20

Also, you can verify that your cluster's nodes do not have external IP addresses with the following command:

1"protoPayload":{
2   "@type":"type.googleapis.com/google.cloud.audit.AuditLog"
3   "authenticationInfo":{
4      "principalEmail":"system:anonymous"
5   }
6   "authorizationInfo":["0":{2}]
7   "methodName":"io.k8s.post"
8   "requestMetadata":{
9      "callerIp":"45.*.*.*"
10      "callerSuppliedUserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
11   }
12   "resourceName":"Autodiscover/Autodiscover.xml"
13   "serviceName":"k8s.io"
14   "status":{
15      "code":"7"
16      "message":"Forbidden"
17   }
18}
19gcloud container clusters describe YOUR_CLUSTER_NAME
20kubectl get nodes --output wide
21

Source https://stackoverflow.com/questions/71605741

QUESTION

Is there a new solution for downloading package and dependencies for a given R version

Asked 2022-Feb-09 at 03:33

I have seen the question asked here from 2018. I'm wondering if there is a better answer today.

Our work computers are bedeviled by an entire IT security department that seems to exist to make them useless. We are allowed to run R 3.6.3 (4.x hasn't been approved yet). We cannot connect to CRAN from behind the corporate firewall. In the past that meant we took our laptops home to install packages. But now we have a download monitor that blocks CRAN downloads even when we're on our own wi-fi.

I was attempting to get around this by downloading the package .zip files on a personal machine, transferring them via CD, and then installing with repos=NULL. I used this code

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9

to get the dependencies for the package I wanted and downloaded them all with download.packages(). I was trying to install tidyquant so it turned out there were 113 dependencies.

But of course this downloads the latest version of all 113 packages, several of which it turned out were incompatible with R 3.6.3.

The solution I referenced above suggested finding a compatible version and then using

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12

But that would involve manually searching all 113 dependencies...plus I'm looking for a general purpose solution that I can share with the ~10 other people in my office that all have this issue now.

I'm hoping that since 2018 a better solution may have presented itself?

Update: Based on answers below I tried

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24

I get

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46

And the "no package" warnings go on for all 113 dependencies.

I think the key here is

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47

Obviously there wouldn't be a 4.1 repository in April 2020. I assume it's trying that since the machine is running 4.1. So it seems we're tantalizingly close....

If it helps

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64

ANSWER

Answered 2022-Feb-09 at 03:33

I'm not sure if it completely addresses your needs, but package checkpoint seems appropriate here. It allows you to download source packages from a snapshot of CRAN taken at a specified date, going back to 2014-09-17. R 4.0.0 was released around 2020-04-24, so the snapshot from 2020-04-01 should work for your purposes.

Here is a reproducible example:

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64tmp1 <- tempfile()
65dir.create(tmp1)
66cwd <- setwd(tmp1)
67
68getOption("repos")
69##                           CRAN 
70## "https://cloud.r-project.org/"
71
72available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
73##         Version   Repository                               
74## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
75## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
76## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"
77
78download.packages("Matrix", ".", type = "source")
79## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
80## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
81## ==================================================
82## downloaded 2.7 MB
83## 
84##      [,1]     [,2]                   
85## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"
86
87tmp2 <- tempfile()
88dir.create(tmp2)
89checkpoint::checkpoint(snapshot_date = "2020-04-01", 
90                       r_version = "3.6.3", 
91                       checkpoint_location = tmp2,
92                       scan_now = FALSE)
93## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
94## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
95## Warning messages:
96## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
97##   Specified R version not the same as current R version
98## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
99##   Specified R version not the same as current R version
100
101getOption("repos")
102##                                             CRAN 
103## "https://mran.microsoft.com/snapshot/2020-04-01"
104
105available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
106##         Version   Repository                                                  
107## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
108## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
109## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
110
111download.packages("Matrix", ".", type = "source")
112## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
113## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
114## ==================================================
115## downloaded 1.8 MB
116## 
117##      [,1]     [,2]                    
118## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"
119
120setwd(cwd)
121unlink(c(tmp1, tmp2), recursive = TRUE)
122

The warnings about version mismatch occur if you are not actually running R 3.6.3. They can be ignored if you are only downloading source packages, with the intention of installing them on another machine actually running 3.6.3.

You can take a look at the package README and ?checkpoint for more details.

Update

If you are trying to download binary packages (.zip for Windows, .tgz for macOS) rather than source packages (.tar.gz), then checkpoint can get you into trouble. By default, download.packages and friends use contrib.url(repos, type) to construct a URL to search for package binaries.

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64tmp1 <- tempfile()
65dir.create(tmp1)
66cwd <- setwd(tmp1)
67
68getOption("repos")
69##                           CRAN 
70## "https://cloud.r-project.org/"
71
72available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
73##         Version   Repository                               
74## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
75## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
76## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"
77
78download.packages("Matrix", ".", type = "source")
79## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
80## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
81## ==================================================
82## downloaded 2.7 MB
83## 
84##      [,1]     [,2]                   
85## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"
86
87tmp2 <- tempfile()
88dir.create(tmp2)
89checkpoint::checkpoint(snapshot_date = "2020-04-01", 
90                       r_version = "3.6.3", 
91                       checkpoint_location = tmp2,
92                       scan_now = FALSE)
93## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
94## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
95## Warning messages:
96## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
97##   Specified R version not the same as current R version
98## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
99##   Specified R version not the same as current R version
100
101getOption("repos")
102##                                             CRAN 
103## "https://mran.microsoft.com/snapshot/2020-04-01"
104
105available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
106##         Version   Repository                                                  
107## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
108## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
109## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
110
111download.packages("Matrix", ".", type = "source")
112## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
113## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
114## ==================================================
115## downloaded 1.8 MB
116## 
117##      [,1]     [,2]                    
118## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"
119
120setwd(cwd)
121unlink(c(tmp1, tmp2), recursive = TRUE)
122contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
123## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"
124
125contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
126## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"
127

But there is nothing at either URL. That is (in part) because contrib.url appends the R version that you are currently running, which might not have existed on your snapshot date. Hence:

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64tmp1 <- tempfile()
65dir.create(tmp1)
66cwd <- setwd(tmp1)
67
68getOption("repos")
69##                           CRAN 
70## "https://cloud.r-project.org/"
71
72available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
73##         Version   Repository                               
74## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
75## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
76## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"
77
78download.packages("Matrix", ".", type = "source")
79## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
80## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
81## ==================================================
82## downloaded 2.7 MB
83## 
84##      [,1]     [,2]                   
85## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"
86
87tmp2 <- tempfile()
88dir.create(tmp2)
89checkpoint::checkpoint(snapshot_date = "2020-04-01", 
90                       r_version = "3.6.3", 
91                       checkpoint_location = tmp2,
92                       scan_now = FALSE)
93## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
94## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
95## Warning messages:
96## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
97##   Specified R version not the same as current R version
98## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
99##   Specified R version not the same as current R version
100
101getOption("repos")
102##                                             CRAN 
103## "https://mran.microsoft.com/snapshot/2020-04-01"
104
105available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
106##         Version   Repository                                                  
107## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
108## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
109## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
110
111download.packages("Matrix", ".", type = "source")
112## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
113## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
114## ==================================================
115## downloaded 1.8 MB
116## 
117##      [,1]     [,2]                    
118## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"
119
120setwd(cwd)
121unlink(c(tmp1, tmp2), recursive = TRUE)
122contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
123## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"
124
125contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
126## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"
127download.packages("Matrix", ".", type = "win.binary")
128## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
129##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
130## Warning in download.packages("Matrix", ".", type = "win.binary") :
131##   no package 'Matrix' at the repositories
132##      [,1] [,2]
133
134download.packages("Matrix", ".", type = "mac.binary")
135## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1:
136##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1/PACKAGES'
137## Warning in download.packages("Matrix", ".", type = "mac.binary") :
138##   no package 'Matrix' at the repositories
139##      [,1] [,2]
140

The URLs that you actually need are:

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64tmp1 <- tempfile()
65dir.create(tmp1)
66cwd <- setwd(tmp1)
67
68getOption("repos")
69##                           CRAN 
70## "https://cloud.r-project.org/"
71
72available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
73##         Version   Repository                               
74## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
75## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
76## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"
77
78download.packages("Matrix", ".", type = "source")
79## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
80## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
81## ==================================================
82## downloaded 2.7 MB
83## 
84##      [,1]     [,2]                   
85## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"
86
87tmp2 <- tempfile()
88dir.create(tmp2)
89checkpoint::checkpoint(snapshot_date = "2020-04-01", 
90                       r_version = "3.6.3", 
91                       checkpoint_location = tmp2,
92                       scan_now = FALSE)
93## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
94## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
95## Warning messages:
96## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
97##   Specified R version not the same as current R version
98## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
99##   Specified R version not the same as current R version
100
101getOption("repos")
102##                                             CRAN 
103## "https://mran.microsoft.com/snapshot/2020-04-01"
104
105available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
106##         Version   Repository                                                  
107## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
108## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
109## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
110
111download.packages("Matrix", ".", type = "source")
112## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
113## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
114## ==================================================
115## downloaded 1.8 MB
116## 
117##      [,1]     [,2]                    
118## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"
119
120setwd(cwd)
121unlink(c(tmp1, tmp2), recursive = TRUE)
122contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
123## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"
124
125contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
126## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"
127download.packages("Matrix", ".", type = "win.binary")
128## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
129##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
130## Warning in download.packages("Matrix", ".", type = "win.binary") :
131##   no package 'Matrix' at the repositories
132##      [,1] [,2]
133
134download.packages("Matrix", ".", type = "mac.binary")
135## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1:
136##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1/PACKAGES'
137## Warning in download.packages("Matrix", ".", type = "mac.binary") :
138##   no package 'Matrix' at the repositories
139##      [,1] [,2]
140## Windows
141"https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"
142
143## macOS, OS X, whatever
144"https://cran.microsoft.com/snapshot/2020-04-01/bin/macosx/el-capitan/contrib/3.6"
145

The best way forward, in this case, may be to avoid checkpoint altogether and directly pass the valid URL to available.packages and download.packages:

1getPackages <- function(packs){
2  packages <- unlist(
3    tools::package_dependencies(packs, available.packages(),
4                         which=c("Depends", "Imports"), recursive=TRUE)
5  )
6  packages <- union(packs, packages)
7  packages
8}
9install_version("<package name>",
10                version = "<version number>",
11                repos = "http://cran.us.r-project.org")
12tmp <- tempfile()
13dir.create(tmp)
14
15checkpoint::checkpoint(snapshot_date = "2020-04-01", 
16                       r_version = "3.6.3", 
17                       checkpoint_location = tmp,
18                       scan_now = FALSE)
19
20
21packages <- getPackages(c("tidyquant"))
22
23download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
24Running create_checkpoint in the home directory may result
25in checkpointing very many packages. Continue? (Y/n) Y
26
27Creating checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
28Using checkpoint directory C:/Users/jerem/AppData/Local/Temp/Rtmpa2YEjU/file1efc6daf58e3/.checkpoint/2020-04-01/lib/x86_64-w64-mingw32/3.6.3
29Warning messages:
301: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
31  Specified R version not the same as current R version
322: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
33  Specified R version not the same as current R version
34> 
35> packages <- getPackages(c("tidyquant"))
36> 
37> download.packages(packages, destdir="C:\\Users\\jerem\\Downloads\\tidyquant", type="win.binary")
38Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
39  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
40Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
41  no package โ€˜tidyquantโ€™ at the repositories
42Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
43  no package โ€˜lubridateโ€™ at the repositories
44Warning in download.packages(packages, destdir = "C:\\Users\\jerem\\Downloads\\tidyquant",  :
45  no package โ€˜PerformanceAnalyticsโ€™ at the repositories
46Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
47> sessionInfo()
48R version 4.1.2 (2021-11-01)
49Platform: x86_64-w64-mingw32/x64 (64-bit)
50Running under: Windows 10 x64 (build 22000)
51
52Matrix products: default
53
54locale:
55[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
56[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
57system code page: 65001
58
59attached base packages:
60[1] stats     graphics  grDevices utils     datasets  methods   base     
61
62loaded via a namespace (and not attached):
63[1] compiler_4.1.2   tools_4.1.2      checkpoint_1.0.2
64tmp1 <- tempfile()
65dir.create(tmp1)
66cwd <- setwd(tmp1)
67
68getOption("repos")
69##                           CRAN 
70## "https://cloud.r-project.org/"
71
72available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
73##         Version   Repository                               
74## lattice "0.20-45" "https://cloud.r-project.org/src/contrib"
75## Matrix  "1.4-0"   "https://cloud.r-project.org/src/contrib"
76## nlme    "3.1-155" "https://cloud.r-project.org/src/contrib"
77
78download.packages("Matrix", ".", type = "source")
79## trying URL 'https://cloud.r-project.org/src/contrib/Matrix_1.4-0.tar.gz'
80## Content type 'application/x-gzip' length 2849865 bytes (2.7 MB)
81## ==================================================
82## downloaded 2.7 MB
83## 
84##      [,1]     [,2]                   
85## [1,] "Matrix" "./Matrix_1.4-0.tar.gz"
86
87tmp2 <- tempfile()
88dir.create(tmp2)
89checkpoint::checkpoint(snapshot_date = "2020-04-01", 
90                       r_version = "3.6.3", 
91                       checkpoint_location = tmp2,
92                       scan_now = FALSE)
93## Creating checkpoint directory /var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T//RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
94## Using checkpoint directory /private/var/folders/n7/v9s56rmd5hn17d3f1qj13l7m0000gn/T/RtmpbrT5Br/filee2045e35c290/.checkpoint/2020-04-01/lib/aarch64-apple-darwin20/3.6.3
95## Warning messages:
96## 1: In create_checkpoint(snapshot_date, r_version, checkpoint_location,  :
97##   Specified R version not the same as current R version
98## 2: In use_checkpoint(snapshot_date, r_version, checkpoint_location,  :
99##   Specified R version not the same as current R version
100
101getOption("repos")
102##                                             CRAN 
103## "https://mran.microsoft.com/snapshot/2020-04-01"
104
105available.packages()[c("lattice", "Matrix", "nlme"), c("Version", "Repository")]
106##         Version   Repository                                                  
107## lattice "0.20-40" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
108## Matrix  "1.2-18"  "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
109## nlme    "3.1-145" "https://mran.microsoft.com/snapshot/2020-04-01/src/contrib"
110
111download.packages("Matrix", ".", type = "source")
112## trying URL 'https://mran.microsoft.com/snapshot/2020-04-01/src/contrib/Matrix_1.2-18.tar.gz'
113## Content type 'application/octet-stream' length 1871705 bytes (1.8 MB)
114## ==================================================
115## downloaded 1.8 MB
116## 
117##      [,1]     [,2]                    
118## [1,] "Matrix" "./Matrix_1.2-18.tar.gz"
119
120setwd(cwd)
121unlink(c(tmp1, tmp2), recursive = TRUE)
122contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "win.binary")
123## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1"
124
125contrib.url("https://mran.microsoft.com/snapshot/2020-04-01/src/contrib", "mac.binary")
126## [1] "https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1"
127download.packages("Matrix", ".", type = "win.binary")
128## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1:
129##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/4.1/PACKAGES'
130## Warning in download.packages("Matrix", ".", type = "win.binary") :
131##   no package 'Matrix' at the repositories
132##      [,1] [,2]
133
134download.packages("Matrix", ".", type = "mac.binary")
135## Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1:
136##   cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-01/bin/macosx/contrib/4.1/PACKAGES'
137## Warning in download.packages("Matrix", ".", type = "mac.binary") :
138##   no package 'Matrix' at the repositories
139##      [,1] [,2]
140## Windows
141"https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"
142
143## macOS, OS X, whatever
144"https://cran.microsoft.com/snapshot/2020-04-01/bin/macosx/el-capitan/contrib/3.6"
145pkg <- "tidyquant"
146contriburl <- "https://cran.microsoft.com/snapshot/2020-04-01/bin/windows/contrib/3.6"
147
148db <- available.packages(contriburl)
149deps <- tools::package_dependencies(pkg, db, recursive = TRUE)[[pkg]]
150download.packages(c(pkg, deps), ".", contriburl = contriburl, type = "win.binary")
151

Source https://stackoverflow.com/questions/71021876

QUESTION

Kubernetes NodePort is not available on all nodes - Oracle Cloud Infrastructure (OCI)

Asked 2022-Jan-31 at 14:37

I've been trying to get over this but I'm out of ideas for now hence I'm posting the question here.

I'm experimenting with the Oracle Cloud Infrastructure (OCI) and I wanted to create a Kubernetes cluster which exposes some service.

The goal is:

  • A running managed Kubernetes cluster (OKE)
  • 2 nodes at least
  • 1 service that's accessible for external parties

The infra looks the following:

  • A VCN for the whole thing
  • A private subnet on 10.0.1.0/24
  • A public subnet on 10.0.0.0/24
  • NAT gateway for the private subnet
  • Internet gateway for the public subnet
  • Service gateway
  • The corresponding security lists for both subnets which I won't share right now unless somebody asks for it
  • A containerengine K8S (OKE) cluster in the VCN with public Kubernetes API enabled
  • A node pool for the K8S cluster with 2 availability domains and with 2 instances right now. The instances are ARM machines with 1 OCPU and 6GB RAM running Oracle-Linux-7.9-aarch64-2021.12.08-0 images.
  • A namespace in the K8S cluster (call it staging for now)
  • A deployment which refers to a custom NextJS application serving traffic on port 3000

And now it's the point where I want to expose the service running on port 3000.

I have 2 obvious choices:

  • Create a LoadBalancer service in K8S which will spawn a classic Load Balancer in OCI, set up it's listener and set up the backendset referring to the 2 nodes in the cluster, plus it adjusts the subnet security lists to make sure traffic can flow
  • Create a Network Load Balancer in OCI and create a NodePort on K8S and manually configure the NLB to the ~same settings as the classic Load Balancer

The first one works perfectly fine but I want to use this cluster with minimal costs so I decided to experiment with option 2, the NLB since it's way cheaper (zero cost).

Long story short, everything works and I can access the NextJS app on the IP of the NLB most of the time but sometimes I couldn't. I decided to look it up what's going on and turned out the NodePort that I exposed in the cluster isn't working how I'd imagine.

The service behind the NodePort is only accessible on the Node that's running the pod in K8S. Assume NodeA is running the service and NodeB is just there chilling. If I try to hit the service on NodeA, everything is fine. But when I try to do the same on NodeB, I don't get a response at all.

That's my problem and I couldn't figure out what could be the issue.

What I've tried so far:

  • Switching from ARM machines to AMD ones - no change
  • Created a bastion host in the public subnet to test which nodes are responding to requests. Turned out only the node responds that's running the pod.
  • Created a regular LoadBalancer in K8S with the same config as the NodePort (in this case OCI will create a classic Load Balancer), that works perfectly
  • Tried upgrading to Oracle 8.4 images for the K8S nodes, didn't fix it
  • Ran the Node Doctor on the nodes, everything is fine
  • Checked the logs of kube-proxy, kube-flannel, core-dns, no error
  • Since the cluster consists of 2 nodes, I gave it a try and added one more node and the service was not accessible on the new node either
  • Recreated the cluster from scratch

Edit: Some update. I've tried to use a DaemonSet instead of a regular Deployment for the pod to ensure that as a temporary solution, all nodes are running at least one instance of the pod and surprise. The node that was previously not responding to requests on that specific port, it still does not, even though a pod is running on it.

Edit2: Originally I was running the latest K8S version for the cluster (v1.21.5) and I tried downgrading to v1.20.11 and unfortunately the issue is still present.

Edit3: Checked if the NodePort is open on the node that's not responding and it is, at least kube-proxy is listening on it.

1tcp        0      0 0.0.0.0:31600           0.0.0.0:*               LISTEN      16671/kube-proxy
2

Edit4:: Tried adding whitelisting iptables rules but didn't change anything.

1tcp        0      0 0.0.0.0:31600           0.0.0.0:*               LISTEN      16671/kube-proxy
2[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P FORWARD ACCEPT
3[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P INPUT ACCEPT
4[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P OUTPUT ACCEPT
5

Edit5: Just as a trial, I created a LoadBalancer once more to verify if I'm gone completely mental and I just didn't notice this error when I tried or it really works. Funny thing, it works perfectly fine through the classic load balancer's IP. But when I try to send a request to the nodes directly on the port that was opened for the load balancer (it's 30679 for now). I get response only from the node that's running the pod. From the other, still nothing yet through the load balancer, I get 100% successful responses.

Bonus, here's the iptables from the Node that's not responding to requests, not too sure what to look for:

1tcp        0      0 0.0.0.0:31600           0.0.0.0:*               LISTEN      16671/kube-proxy
2[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P FORWARD ACCEPT
3[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P INPUT ACCEPT
4[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P OUTPUT ACCEPT
5[opc@oke-cn44eyuqdoq-n3ewna4fqra-sx5p5dalkuq-1 ~]$ sudo iptables -L
6Chain INPUT (policy ACCEPT)
7target     prot opt source               destination
8KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes health check service ports */
9KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
10KUBE-FIREWALL  all  --  anywhere             anywhere
11
12Chain FORWARD (policy ACCEPT)
13target     prot opt source               destination
14KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
15KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
16KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
17ACCEPT     all  --  10.244.0.0/16        anywhere
18ACCEPT     all  --  anywhere             10.244.0.0/16
19
20Chain OUTPUT (policy ACCEPT)
21target     prot opt source               destination
22KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
23KUBE-FIREWALL  all  --  anywhere             anywhere
24
25Chain KUBE-EXTERNAL-SERVICES (2 references)
26target     prot opt source               destination
27
28Chain KUBE-FIREWALL (2 references)
29target     prot opt source               destination
30DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
31DROP       all  -- !loopback/8           loopback/8           /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
32
33Chain KUBE-FORWARD (1 references)
34target     prot opt source               destination
35DROP       all  --  anywhere             anywhere             ctstate INVALID
36ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
37ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
38ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
39
40Chain KUBE-KUBELET-CANARY (0 references)
41target     prot opt source               destination
42
43Chain KUBE-NODEPORTS (1 references)
44target     prot opt source               destination
45
46Chain KUBE-PROXY-CANARY (0 references)
47target     prot opt source               destination
48
49Chain KUBE-SERVICES (2 references)
50target     prot opt source               destination
51

Service spec (the running one since it was generated using Terraform):

1tcp        0      0 0.0.0.0:31600           0.0.0.0:*               LISTEN      16671/kube-proxy
2[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P FORWARD ACCEPT
3[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P INPUT ACCEPT
4[opc@oke-cdvpd5qrofa-nyx7mjtqw4a-svceq4qaiwq-0 ~]$ sudo iptables -P OUTPUT ACCEPT
5[opc@oke-cn44eyuqdoq-n3ewna4fqra-sx5p5dalkuq-1 ~]$ sudo iptables -L
6Chain INPUT (policy ACCEPT)
7target     prot opt source               destination
8KUBE-NODEPORTS  all  --  anywhere             anywhere             /* kubernetes health check service ports */
9KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
10KUBE-FIREWALL  all  --  anywhere             anywhere
11
12Chain FORWARD (policy ACCEPT)
13target     prot opt source               destination
14KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
15KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
16KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
17ACCEPT     all  --  10.244.0.0/16        anywhere
18ACCEPT     all  --  anywhere             10.244.0.0/16
19
20Chain OUTPUT (policy ACCEPT)
21target     prot opt source               destination
22KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
23KUBE-FIREWALL  all  --  anywhere             anywhere
24
25Chain KUBE-EXTERNAL-SERVICES (2 references)
26target     prot opt source               destination
27
28Chain KUBE-FIREWALL (2 references)
29target     prot opt source               destination
30DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
31DROP       all  -- !loopback/8           loopback/8           /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
32
33Chain KUBE-FORWARD (1 references)
34target     prot opt source               destination
35DROP       all  --  anywhere             anywhere             ctstate INVALID
36ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
37ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
38ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
39
40Chain KUBE-KUBELET-CANARY (0 references)
41target     prot opt source               destination
42
43Chain KUBE-NODEPORTS (1 references)
44target     prot opt source               destination
45
46Chain KUBE-PROXY-CANARY (0 references)
47target     prot opt source               destination
48
49Chain KUBE-SERVICES (2 references)
50target     prot opt source               destination
51{
52    "apiVersion": "v1",
53    "kind": "Service",
54    "metadata": {
55        "creationTimestamp": "2022-01-28T09:13:33Z",
56        "name": "web-staging-service",
57        "namespace": "web-staging",
58        "resourceVersion": "22542",
59        "uid": "c092f99b-7c72-4c32-bf27-ccfa1fe92a79"
60    },
61    "spec": {
62        "clusterIP": "10.96.99.112",
63        "clusterIPs": [
64            "10.96.99.112"
65        ],
66        "externalTrafficPolicy": "Cluster",
67        "ipFamilies": [
68            "IPv4"
69        ],
70        "ipFamilyPolicy": "SingleStack",
71        "ports": [
72            {
73                "nodePort": 31600,
74                "port": 3000,
75                "protocol": "TCP",
76                "targetPort": 3000
77            }
78        ],
79        "selector": {
80            "app": "frontend"
81        },
82        "sessionAffinity": "None",
83        "type": "NodePort"
84    },
85    "status": {
86        "loadBalancer": {}
87    }
88}
89

Any ideas are appreciated. Thanks guys.

ANSWER

Answered 2022-Jan-31 at 12:06

Might not be the ideal fix, but can you try changing the externalTrafficPolicy to Local. This would prevent the health check on the nodes which don't run the application to fail. This way the traffic will only be forwarded to the node where the application is . Setting externalTrafficPolicy to local is also a requirement to preserve source IP of the connection. Also, can you share the health check config for both NLB and LB that you are using. When you change the externalTrafficPolicy, note that the health check for LB would change and the same needs to be applied to NLB.

Edit: Also note that you need a security list/ network security group added to your node subnet/nodepool, which allows traffic on all protocols from the worker node subnet.

Source https://stackoverflow.com/questions/70893487

QUESTION

How can we mount a file as read-only in Linux through Go?

Asked 2022-Jan-25 at 16:08

I want to know how can we mount an file as read-only in Linux CentOS 7 Server through Golang. I have tried syscall but that doesn't work, syscall mounts the file but as read-write i have tried to give ro argument in the data but still it's mounting as read-write. Here is my go code:

1syscall.Mount(src, dst, "auto", syscall.MS_BIND, "ro")
2

You can see I have given ro argument in the data, i have also tried to give only r and readonly and also read-only but none of them works, when i compile the go file and execute it and then when i check /etc/mtab then i am getting this output :

1syscall.Mount(src, dst, "auto", syscall.MS_BIND, "ro")
2cat /etc/mtab | grep "firewall"
3/dev/vda1 /root/firewall.txt ext4 rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=quota.user 0 0
4

Here you can see that firewall.txt is mounted as rw means read-write

I want to mount it as read-only , can anyone help me how can i do that in Golang?

ANSWER

Answered 2022-Jan-25 at 14:23

Read-only mode is defined by the syscall flag MS_RDONLY, which is also defined in the syscall package. So the call should be:

1syscall.Mount(src, dst, "auto", syscall.MS_BIND, "ro")
2cat /etc/mtab | grep "firewall"
3/dev/vda1 /root/firewall.txt ext4 rw,relatime,data=ordered,jqfmt=vfsv1,usrjquota=quota.user 0 0
4syscall.Mount(src, dst, "auto", syscall.MS_BIND | syscall.MS_RDONLY, "")
5

Source https://stackoverflow.com/questions/70850199

QUESTION

Terraform: How to modify a public subnet's route table that was created by module 'vpc'?

Asked 2022-Jan-23 at 04:56

I used the vpc module to create my VPC via the following code:

1module "vpc" {
2  source = "terraform-aws-modules/vpc/aws"
3  name = "${var.namespace}-vpc"
4  cidr = "10.0.0.0/16"
5  azs = data.aws_availability_zones.available.names
6  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
7  public_subnets = ["10.0.101.0/24", "10.0.102.0/24"]
8  #assign_generated_ipv6_cidr_block = true
9  create_database_subnet_group     = true
10  enable_nat_gateway               = true
11  single_nat_gateway               = true
12  enable_dns_hostnames = true
13  enable_dns_support   = true
14}
15

This module automatically creates two public subnets that has a route table that points to an internet gateway. However, I would like to modify one of the two public subnets to have a different route table that points to a firewall that I have created.

What I did was to create a new route table pub_to_firewall, and then create a new aws_route_table_association to associate the public subnet with the new route table.

1module "vpc" {
2  source = "terraform-aws-modules/vpc/aws"
3  name = "${var.namespace}-vpc"
4  cidr = "10.0.0.0/16"
5  azs = data.aws_availability_zones.available.names
6  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
7  public_subnets = ["10.0.101.0/24", "10.0.102.0/24"]
8  #assign_generated_ipv6_cidr_block = true
9  create_database_subnet_group     = true
10  enable_nat_gateway               = true
11  single_nat_gateway               = true
12  enable_dns_hostnames = true
13  enable_dns_support   = true
14}
15resource "aws_route_table_association" "sn_to_fw_rt_association" {
16  subnet_id      = module.vpc.public_subnets[0]
17  route_table_id = aws_route_table.pub_to_firewall.id
18  depends_on = [
19    aws_route_table.pub_to_firewall,
20  ]
21}
22

I have been able to follow the instructions to import the original association to this new association, and terraform apply to get the public subnet to have this new route table containing the firewall reference.

However, when I run terraform apply again, terraform now wants to go back to the 'default' associations:

1module "vpc" {
2  source = "terraform-aws-modules/vpc/aws"
3  name = "${var.namespace}-vpc"
4  cidr = "10.0.0.0/16"
5  azs = data.aws_availability_zones.available.names
6  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
7  public_subnets = ["10.0.101.0/24", "10.0.102.0/24"]
8  #assign_generated_ipv6_cidr_block = true
9  create_database_subnet_group     = true
10  enable_nat_gateway               = true
11  single_nat_gateway               = true
12  enable_dns_hostnames = true
13  enable_dns_support   = true
14}
15resource "aws_route_table_association" "sn_to_fw_rt_association" {
16  subnet_id      = module.vpc.public_subnets[0]
17  route_table_id = aws_route_table.pub_to_firewall.id
18  depends_on = [
19    aws_route_table.pub_to_firewall,
20  ]
21}
22Objects have changed outside of Terraform
23
24Terraform detected the following changes made outside of Terraform since the last "terraform apply":
25
26  # module.networking.module.vpc.aws_route_table_association.public[0] has been deleted
27  - resource "aws_route_table_association" "public" {
28      - id             = "rtbassoc-[ ]" -> null
29      - route_table_id = "rtb-0cabc2388adXXXXX" -> null
30      - subnet_id      = "subnet-0a2b011cd7aXXXXX" -> null
31    }
32
33
34Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these
35changes.
36โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
37
38Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
39  + create
40  ~ update in-place
41
42Terraform will perform the following actions:
43
44
45  # module.networking.module.vpc.aws_route_table_association.public[0] will be created
46  + resource "aws_route_table_association" "public" {
47      + id             = (known after apply)
48      + route_table_id = "rtb-0cabc2388adXXXXX"
49      + subnet_id      = "subnet-0a2b011cd73XXXXX"
50    }
51

I do not want this resource to be recreated because it would throw an error that โ”‚ Error: error creating Route Table (rtb-0cabc2388adXXXXX) Association: Resource.AlreadyAssociated: the specified association for route table rtb-0cabc2388adXXXXX conflicts with an existing association obviously since I already associated it with the new routing table.

How can I either:

  1. Force terraform to 'ignore' the default subnet to routing tables setup
  2. Or update the vpc created aws_route_table_association resource module.networking.module.vpc.aws_route_table_association.public[0] to reference the new route table instead?

ANSWER

Answered 2022-Jan-21 at 09:05

You can't change that, as this is how the aws vpc module works. You need custom designed VPC for that. So you have to either fork the entire module and made the changes that you want, or create new VPC module from scratch tailored to your needs.

Source https://stackoverflow.com/questions/70798260

QUESTION

Why is ArgoCD confusing GitHub.com with my own public IP?

Asked 2022-Jan-10 at 17:37

I have just set up a kubernetes cluster on bare metal using kubeadm, Flannel and MetalLB. Next step for me is to install ArgoCD.

I installed the ArgoCD yaml from the "Getting Started" page and logged in.

When adding my Git repositories ArgoCD gives me very weird error messages: enter image description here The error message seems to suggest that ArgoCD for some reason is resolving github.com to my public IP address (I am not exposing SSH, therefore connection refused).

I can not find any reason why it would do this. When using https:// instead of SSH I get the same result, but on port 443.

I have put a dummy pod in the same namespace as ArgoCD and made some DNS queries. These queries resolved correctly.

What makes ArgoCD think that github.com resolves to my public IP address?

EDIT:

I have also checked for network policies in the argocd namespace and found no policy that was restricting egress.

I have had this working on clusters in the same network previously and have not changed my router firewall since then.

ANSWER

Answered 2022-Jan-08 at 21:04

That looks like argoproj/argo-cd issue 1510, where the initial diagnostic was that the cluster is blocking outbound connections to GitHub. And it suggested to check the egress configuration.

Yet, the issue was resolved with an ingress rule configuration:

need to define in values.yaml.
argo-cd default provide subdomain but in our case it was /argocd

1ingress:
2  enabled: true
3  annotations:
4    kubernetes.io/ingress.class: nginx
5    nginx.ingress.kubernetes.io/backend-protocol: HTTP
6    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
7    nginx.ingress.kubernetes.io/rewrite-target: /
8  path: /argocd
9  hosts:
10    - www.example.com
11

and this I have defined under templates >> argocd-server-deployment.yaml

1ingress:
2  enabled: true
3  annotations:
4    kubernetes.io/ingress.class: nginx
5    nginx.ingress.kubernetes.io/backend-protocol: HTTP
6    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
7    nginx.ingress.kubernetes.io/rewrite-target: /
8  path: /argocd
9  hosts:
10    - www.example.com
11containers: 
12  - name: argocd-server 
13    image: {{ .Values.server.image.repository }}:{{ .Values.server.image.tag }} 
14    imagePullPolicy: {{ .Values.server.image.pullPolicy }} 
15    command: 
16      - argocd-server 
17      - --staticassets - /shared/app - --repo-server - argocd-repo-server:8081 - --insecure - --basehref - /argocd
18

The same case includes an instance very similar to yours:

https://user-images.githubusercontent.com/48730712/92177232-b9839900-ee5d-11ea-844e-9a20c8e8ee4b.png

In any case, do check your git configuration (git config -l) as seen in the ArgoCD cluster, to look for any insteadOf which would change automatically github.com into a local URL (as seen here)

Source https://stackoverflow.com/questions/70600322

QUESTION

New-CimSession without elevation by providing admin credentials on Windows 10?

Asked 2021-Dec-31 at 13:43

I need to query some WMI values using PowerShell from Windows 10 devices. The script is executed in the context of a non-admin user by some software distribution tooling.

There is a local admin account, and for the current purpose (retrieving information before wiping the system) it wouldn't be a problem to put the password in the script. As automation is a hard requirement, there is no way to deal with UAC windows or the user to enter some credentials.

Is there any way to get

1$sess = New-CimSession -Credential $admincred
2

to work without running into Access is denied, because it isn't run in an elevated context? Can I somehow self-elevate it by just having the admin credentials?

[Edit]

The comments asked to provide more concrete information:

I want to onboard many unmanaged (i.e. no software distribution tool, no domain join) Windows 10 devices to Windows Autopilot.

  • The devices are not at a specific site.

  • The device vendor can't provide the information.

  • The users don't have administrative privileges

  • The users don't know the local admin password (I do)

  • Exposing the local admin password is less of a problem than the missing tech knowledge of the users (the password is considered legacy)

  • The firewall is preventing incoming traffic (no RDP, WinRM)

  • Code (Source):

    $devDetail = (Get-CimInstance -CimSession $session -Namespace root/cimv2/mdm/dmmap -Class MDM_DevDetail_Ext01 -Filter "InstanceID='Ext' AND ParentID='./DevDetail'")

It is too time consuming to get the information using manual remote sessions with a tool like Teamviewer. Getting the users to download a tool from the intranet and running it would be a way to go. So I created a standalone application that builds and runs a customized PowerShell script. What won't work is getting it to run in an elevated session. I always end up with Access denied.

ANSWER

Answered 2021-Dec-31 at 13:43

Can I somehow self-elevate it by just having the admin credentials?

No you cannot. UAC is designed to prevent exactly what you are trying to do. Related Q&A:

There may be many workarounds, but they all will have in common that you have to go to your machines (locally or remotely) at least once, gain administrative privileges and prepare something, e. g.:

  • A scheduled task that runs under your local administrator account or under SYSTEM and triggers the execution of your script
  • Disabling UAC (temporarily) (not recommended either way)
  • Installing any remote management software, services or accounts (with extra run as background job privilege)

Source https://stackoverflow.com/questions/70425086

QUESTION

Failing to run Tomcat in IntelliJ IDEA

Asked 2021-Nov-09 at 07:09

I'm trying to use IntelliJ IDEA to run a Tomcat server but failing to so, I keep getting the following error code:

Application Server was not connected before run configuration stop, reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset]

As seen in the following picture: https://i.stack.imgur.com/yqYav.png

Also, I manage to run a Tomcat server in CMD, but it just doesn't seem to work in IntelliJ. This is a pic of me running Tomcat in CMD succesfully and trying to connect to the same port in the IntelliJ (obviously failing to do so) but it does seem to be trying to connect, as in there is nothing really blocking the connection: https://i.stack.imgur.com/OYIvK.png

I tried every solution I could find:

  • specify the same port for both JMX and RMI conversation (here and here)
  • configure the Tomcat server again (here)
  • make the the path to the Tomcat files is correct (here)
  • made sure no Firewall/antivirus is blocking the network connection.
  • uninstall and install again both IntelliJ and Tomcat
  • specify the IP (here)
  • using different ports

Thanks!

ANSWER

Answered 2021-Nov-09 at 07:09

The problem was solved.

Running the Apache Tomcat Installer instead of just downloading the zip solved it.

Source https://stackoverflow.com/questions/69807878

Community Discussions contain sources that include Stack Exchange Network

Tutorials and Learning Resources in Firewall

Tutorials and Learning Resources are not available at this moment for Firewall

Share this Page

share link

Get latest updates on Firewall