Explore all IOT open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in IOT

No Popular Releases are available at this moment for IOT

Popular Libraries in IOT

No Trending Libraries are available at this moment for IOT

Trending New libraries in IOT

No Trending Libraries are available at this moment for IOT

Top Authors in IOT

No Top Authors are available at this moment for IOT.

Trending Kits in IOT

No Trending Kits are available at this moment for IOT

Trending Discussions on IOT

Azure IoT Central architecture - how are Digital Twins implemented and managed?

Cloud2Edge package: error connectivity:connection.id.enforcement.failed

Connecting Waveshare LCD1602 RGB to Raspberry using C#

Why does the error "Unable to execute HTTP request" occur after unregistering ConnectivityManager.NetworkCallback?

Inject host's environment variable into K8s Pod

Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas

Generate all possible (public internet) IPV4 Address combinations in Python

Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres

DRF testing - create object with a many-to-many relation

Postgres March stange results

Azure IoT Central architecture - how are Digital Twins implemented and managed?
Cloud2Edge package: error connectivity:connection.id.enforcement.failed
Connecting Waveshare LCD1602 RGB to Raspberry using C#
Why does the error "Unable to execute HTTP request" occur after unregistering ConnectivityManager.NetworkCallback?
Inject host's environment variable into K8s Pod
Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas
Generate all possible (public internet) IPV4 Address combinations in Python
Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres
DRF testing - create object with a many-to-many relation
Postgres March stange results

QUESTION

Azure IoT Central architecture - how are Digital Twins implemented and managed?

Asked 2022-Mar-17 at 01:54

I have a system with IoT Hub to ingests events from devices and Device Provisioning Service to provision devices. IoT Hub C# SDKs are used for the management of device tags and desired properties (IoT Hub device twins), and to invoke direct methods or schedule jobs.

Recently I've been experimenting with Azure IoT Central. While I don't plan to use it, I've found Digital Twins (that are being used on Azure IoT Central) to offer a very good way of managing IoT devices and I would like to emulate the same kind of functionality and capabilities on my IoT system.

The high-level architecture of IoT Central does not seem to indicate the services or logic used to manage Digital Twins.

IoT Central high level architecture.

As far as I understand, there are two ways you can start using Digital Twins:

  • Plug and Play Digital Twins
  • Azure Digital Twins service

Question - is Azure IoT Central purely based on Plug and Play Digital Twins and/or does it also use Azure Digital Twins service?

ANSWER

Answered 2022-Mar-02 at 08:02

Yes, IOT Central is purely based on plug and play Digital Twins

plug and play Digital Twins enables solution builders to integrate IoT devices with their solutions without any manual configuration.

Azure Digital Twins can be used to design a digital twin architecture that represents actual IoT devices in a wider cloud solution, and which connects to IoT Hub device twins to send and receive live data.

Reference link: https://docs.microsoft.com/en-us/azure/iot-develop/overview-iot-plug-and-play https://docs.microsoft.com/en-us/azure/digital-twins/overview#:~:text=What%20is%20Azure%20Digital%20Twins%3F%201%20Azure%20Digital,solution.%203%20Service%20limits.%20...%204%20Terminology.

Source https://stackoverflow.com/questions/71224342

QUESTION

Cloud2Edge package: error connectivity:connection.id.enforcement.failed

Asked 2022-Feb-14 at 07:56

I'm testing a deployment of the Eclipse IoT Cloud2Edge package and have followed the instructions here https://www.eclipse.org/packages/packages/cloud2edge/tour/ to test. After creating the new tenant and device, and configuring the connection between Hono and Ditto, I can send telemetry to the new device via the Hono http adapter as shown here:

1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2  "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3  "headers": {},
4  "path": "/features/temperature/properties/value",
5  "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
10

and expected to see this property value updated in Ditto. The updated device property value does not update in Ditto, and when I check the Ditto logs I see the following entries:

1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2  "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3  "headers": {},
4  "path": "/features/temperature/properties/value",
5  "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
102022-02-13 20:11:35,265 INFO  [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Received message from AMQP 1.0 with externalMessageHeaders: {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
112022-02-13 20:11:35,271 INFO  [81c41f10-4d59-435b-8ae1-bf5194dcf6bf] o.e.d.c.s.m.InboundDispatchingSink  - onMapped mappedHeaders ImmutableDittoHeaders [{ditto-entity-id=thing:my-tenant:org.acme:my-device-1, ditto-inbound-payload-mapper=default, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=81c41f10-4d59-435b-8ae1-bf5194dcf6bf}]
122022-02-13 20:11:35,278 INFO  [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink  - Got exception <connectivity:connection.id.enforcement.failed> when processing external message with mapper <default>: <The configured filters could not be matched against the given target with ID 'org.acme:my-device-1'.>
132022-02-13 20:11:35,278 INFO  [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink  - Resolved mapped headers of ImmutableDittoHeaders [{ditto-inbound-payload-mapper=default, ditto-entity-id=thing:my-tenant:org.acme:my-device-1, response-required=false, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=b3b11410-6df8-4bfc-a940-fafa87d65be2}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={hono-device-id={{ header:device_id }}, content-type={{ header:content-type }}}]] : and external headers {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
142022-02-13 20:11:35,283 INFO  [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Acking <ID:AMQP_NO_PREFIX:GenericSenderLink-12> with original external message headers=<{orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}>, isSuccess=<true>, ackType=<1 accepted>
15

I think the problem is the "connectivity:connection.id.enforcement.failed" error but I don't know how to troubleshoot. Any advice appreciated.

ANSWER

Answered 2022-Feb-14 at 07:56

What you configured is the Connection source enforcement which makes sure that a Hono device (identified via the AMQP header device_id) may only updates the twin with the same "thing id" in Ditto.

That enforcement fails as your thingId you set in the Ditto Protocol JSON is my-tenant:org.acme:my-device-1 - the topic's first segment is the namespace, the second segment the name - combined those 2 segments become the "thing ID", see also: Protocol topic specification.

So you probably want to send the following message instead:

1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2  "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3  "headers": {},
4  "path": "/features/temperature/properties/value",
5  "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
102022-02-13 20:11:35,265 INFO  [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Received message from AMQP 1.0 with externalMessageHeaders: {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
112022-02-13 20:11:35,271 INFO  [81c41f10-4d59-435b-8ae1-bf5194dcf6bf] o.e.d.c.s.m.InboundDispatchingSink  - onMapped mappedHeaders ImmutableDittoHeaders [{ditto-entity-id=thing:my-tenant:org.acme:my-device-1, ditto-inbound-payload-mapper=default, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=81c41f10-4d59-435b-8ae1-bf5194dcf6bf}]
122022-02-13 20:11:35,278 INFO  [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink  - Got exception <connectivity:connection.id.enforcement.failed> when processing external message with mapper <default>: <The configured filters could not be matched against the given target with ID 'org.acme:my-device-1'.>
132022-02-13 20:11:35,278 INFO  [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink  - Resolved mapped headers of ImmutableDittoHeaders [{ditto-inbound-payload-mapper=default, ditto-entity-id=thing:my-tenant:org.acme:my-device-1, response-required=false, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=b3b11410-6df8-4bfc-a940-fafa87d65be2}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={hono-device-id={{ header:device_id }}, content-type={{ header:content-type }}}]] : and external headers {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
142022-02-13 20:11:35,283 INFO  [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Acking <ID:AMQP_NO_PREFIX:GenericSenderLink-12> with original external message headers=<{orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}>, isSuccess=<true>, ackType=<1 accepted>
15{
16  "topic": "org.acme/my-device-1/things/twin/commands/modify",
17...
18}
19

Source https://stackoverflow.com/questions/71104483

QUESTION

Connecting Waveshare LCD1602 RGB to Raspberry using C#

Asked 2022-Jan-01 at 18:49

I am trying to connect LCD1602 RBG Waveshare to the Raspberry using C#. I connected it to the Raspberry and set the permissions, now trying to pass some data. The code below run all lines, but the LCD is not reacting. If anyone can advise me.

1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11    public class Program
12    {
13        static void Main(string[] args)
14        {
15            Console.WriteLine("Test1");
16
17            using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18            //BCM2835
19            using var driver = new Pcf8574(i2c);
20            using var lcd = new Lcd1602(registerSelectPin: 0,
21                                    enablePin: 2,
22                                    dataPins: new int[] { 4, 5, 6, 7 },
23                                    backlightPin: 3,
24                                    backlightBrightness: 0.3f,
25                                    readWritePin: 1,
26                                    controller: new GpioController(PinNumberingScheme.Logical, driver));
27            int counter = 0;
28
29            while (counter <= 4)
30            {
31                lcd.Clear();
32
33                lcd.SetCursorPosition(0, 0);
34                lcd.Write("TestText1");
35
36                lcd.SetCursorPosition(0, 1);
37                lcd.Write("TestText2");
38
39                Thread.Sleep(2000);
40                counter++;
41            }
42        }
43    }
44}
45
1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11    public class Program
12    {
13        static void Main(string[] args)
14        {
15            Console.WriteLine("Test1");
16
17            using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18            //BCM2835
19            using var driver = new Pcf8574(i2c);
20            using var lcd = new Lcd1602(registerSelectPin: 0,
21                                    enablePin: 2,
22                                    dataPins: new int[] { 4, 5, 6, 7 },
23                                    backlightPin: 3,
24                                    backlightBrightness: 0.3f,
25                                    readWritePin: 1,
26                                    controller: new GpioController(PinNumberingScheme.Logical, driver));
27            int counter = 0;
28
29            while (counter <= 4)
30            {
31                lcd.Clear();
32
33                lcd.SetCursorPosition(0, 0);
34                lcd.Write("TestText1");
35
36                lcd.SetCursorPosition(0, 1);
37                lcd.Write("TestText2");
38
39                Thread.Sleep(2000);
40                counter++;
41            }
42        }
43    }
44}
45     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
4600:                         -- -- -- -- -- -- -- --
4710: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4820: 20 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4930: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- 3e --
5040: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5150: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5260: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5370: 70 -- -- -- -- -- -- --
54

enter image description here

ANSWER

Answered 2022-Jan-01 at 18:49

The controller for these displays is normally a HD44780 or compatible. If the python sample works fine, then this should do:

1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11    public class Program
12    {
13        static void Main(string[] args)
14        {
15            Console.WriteLine("Test1");
16
17            using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18            //BCM2835
19            using var driver = new Pcf8574(i2c);
20            using var lcd = new Lcd1602(registerSelectPin: 0,
21                                    enablePin: 2,
22                                    dataPins: new int[] { 4, 5, 6, 7 },
23                                    backlightPin: 3,
24                                    backlightBrightness: 0.3f,
25                                    readWritePin: 1,
26                                    controller: new GpioController(PinNumberingScheme.Logical, driver));
27            int counter = 0;
28
29            while (counter <= 4)
30            {
31                lcd.Clear();
32
33                lcd.SetCursorPosition(0, 0);
34                lcd.Write("TestText1");
35
36                lcd.SetCursorPosition(0, 1);
37                lcd.Write("TestText2");
38
39                Thread.Sleep(2000);
40                counter++;
41            }
42        }
43    }
44}
45     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
4600:                         -- -- -- -- -- -- -- --
4710: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4820: 20 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4930: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- 3e --
5040: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5150: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5260: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5370: 70 -- -- -- -- -- -- --
54var i2cLcdDevice = I2cDevice.Create(new I2cConnectionSettings(busId: 1, deviceAddress: 0x3E));
55var i2cRgbDevice = I2cDevice.Create(new I2cConnectionSettings(busId: 1, deviceAddress: 0x60));
56using LcdRgb lcd = new LcdRgb(new Size(16, 2), i2cLcdDevice, i2cRgbDevice);
57{
58    lcd.Write("Hello World!");
59    lcd.SetBacklightColor(Color.Azure);
60}
61

(for reasons beyond me, the python sample defines the addresses as 0x7c >> 1 and 0xc0 >> 1, which are 0x3E and 0x60, which aligns with your device scan)

Source https://stackoverflow.com/questions/70536639

QUESTION

Why does the error "Unable to execute HTTP request" occur after unregistering ConnectivityManager.NetworkCallback?

Asked 2021-Dec-07 at 17:48

In an Android app which uses AWS services, if I deregister a registered ConnectivityManager.NetworkCallback, the app can no longer contact AWS services. I am uncertain why this is occurring, or how to contact the AWS services again. Currently, the only way to reconnect to AWS is to terminate the app and restart it.

To elaborate, one function of the app is to connect the user's Android device to a different WiFi network hotspot for the purposes of setting up an IoT device. At this moment of use, the user would be logged in using AWS's Cognito service. Because how an app can connect to WiFi was changed starting with API level 29, this associated code is only invoked on such devices and the problem is isolated to said devices. Here is the relevant snippet for how the connection is being created:

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18

Unregistering the callback occurs if the setup is completed, if an error is encountered, or if the user decides to navigate backwards out of this setup. Regardless, this is the code used to accomplish unregistering:

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19        .getApplication()
20        .getApplicationContext()
21        .getSystemService(Context.CONNECTIVITY_SERVICE);
22    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23    if (connectivityManager != null) {
24        try {
25            connectivityManager.unregisterNetworkCallback(networkCallback);
26        } catch (IllegalArgumentException e) {
27            // Network is already unregistered.
28            e.printStackTrace();
29        }
30    }
31

This code successfully invokes in all circumstances. At this point, the user is no longer connected to the WiFi hotspot. In my testing, I would now be connected back to my home WiFi with my mobile network available too. However, the app can no longer contact AWS. Considering that my device is still connected to a WiFi network AND my mobile network, I would not expect this outcome. Here is an example stack trace for trying to contact Cognito (although the same sort of error occurs if DynamoDB is contacted, for example):

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19        .getApplication()
20        .getApplicationContext()
21        .getSystemService(Context.CONNECTIVITY_SERVICE);
22    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23    if (connectivityManager != null) {
24        try {
25            connectivityManager.unregisterNetworkCallback(networkCallback);
26        } catch (IllegalArgumentException e) {
27            // Network is already unregistered.
28            e.printStackTrace();
29        }
30    }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err:     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err:     at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err:     at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err:     at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err:     at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err:     at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err:     at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err:     at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err:     at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err:     at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err:     at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err:     at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err:     at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err:     at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err:     at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err:   ... 7 more
62

I have isolated it to specifically when the app performs .unregisterNetworkCallback() to be the cause. If this is not invoked, then the app will not lose the ability to contact AWS.

Additionally, when attempting any AWS connection after unregistering, many of these statements will print into the log before the exception above is logged:

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19        .getApplication()
20        .getApplicationContext()
21        .getSystemService(Context.CONNECTIVITY_SERVICE);
22    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23    if (connectivityManager != null) {
24        try {
25            connectivityManager.unregisterNetworkCallback(networkCallback);
26        } catch (IllegalArgumentException e) {
27            // Network is already unregistered.
28            e.printStackTrace();
29        }
30    }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err:     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err:     at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err:     at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err:     at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err:     at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err:     at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err:     at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err:     at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err:     at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err:     at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err:     at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err:     at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err:     at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err:     at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err:     at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err:   ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63

In my research, it seems to have something to do with Samsung devices. I am testing this on a Samsung Galaxy S10 Lite. I am not sure if this is related to the issue at hand.

Why is this occurring? Am I somehow using the ConnectivityManager.NetworkCallback incorrectly, either in registering or unregistering it? Is it an issue with AWS?

ANSWER

Answered 2021-Dec-07 at 17:48

I realized the issue involved a snippet of code outside of the code shared in this question. The NetworkCallback's onAvailable() is performed as such:

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19        .getApplication()
20        .getApplicationContext()
21        .getSystemService(Context.CONNECTIVITY_SERVICE);
22    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23    if (connectivityManager != null) {
24        try {
25            connectivityManager.unregisterNetworkCallback(networkCallback);
26        } catch (IllegalArgumentException e) {
27            // Network is already unregistered.
28            e.printStackTrace();
29        }
30    }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err:     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err:     at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err:     at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err:     at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err:     at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err:     at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err:     at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err:     at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err:     at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err:     at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err:     at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err:     at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err:     at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err:     at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err:     at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err:   ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63    @Override
64    public void onAvailable(@NonNull Network network) {
65        connectivityManager.bindProcessToNetwork(network);
66    }
67

The reason we are calling .bindProcessToNetwork() is so that the app can still access the internet beyond just the WiFi network.

This is key to why the app is losing connection to AWS after calling .unregisterNetworkCallback(); it is being unregistered but it is still bound to the network we just unregistered. Therefore, when one goes to unregister the NetworkCallback, be sure to first invoke .bindProcessToNetwork() and pass null to clear out the binding:

1    NetworkRequest request =
2        new NetworkRequest.Builder()
3                .addTransportType(TRANSPORT_WIFI)
4                .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5                .setNetworkSpecifier(specifier)
6                .build();
7
8    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9            .getApplication()
10            .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12    ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14    if (connectivityManager != null && request != null) {
15        // Request the network.
16        connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17    }
18    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19        .getApplication()
20        .getApplicationContext()
21        .getSystemService(Context.CONNECTIVITY_SERVICE);
22    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23    if (connectivityManager != null) {
24        try {
25            connectivityManager.unregisterNetworkCallback(networkCallback);
26        } catch (IllegalArgumentException e) {
27            // Network is already unregistered.
28            e.printStackTrace();
29        }
30    }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err:     at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err:     at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err:     at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err:     at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err:     at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err:     at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err:     at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err:     at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err:     at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err:     at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err:     at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err:     at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err:     at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err:     at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err:     at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err:     at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err:     at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err:     at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err:     at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err:     at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err:   ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63    @Override
64    public void onAvailable(@NonNull Network network) {
65        connectivityManager.bindProcessToNetwork(network);
66    }
67    ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
68        .getApplication()
69        .getApplicationContext()
70        .getSystemService(Context.CONNECTIVITY_SERVICE);
71    WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
72    if (connectivityManager != null) {
73        try {
74            // Remove the app from the network callback first before deregistering it. Otherwise,
75            // the app will be stuck on the deregistered network (giving it no connectivity).
76            connectivityManager.bindProcessToNetwork(null);
77            connectivityManager.unregisterNetworkCallback(networkCallback);
78        } catch (IllegalArgumentException e) {
79            // Network is already unregistered.
80            e.printStackTrace();
81        }
82    }
83

I came to this answer by reading this answer on a different question. It gives a great overview of how the whole WiFi connection/management process works for Android API 29 and higher, including this final part of disconnecting and allowing the user to continue with the app experience on their normal WiFi or cellular network.

Source https://stackoverflow.com/questions/69441778

QUESTION

Inject host's environment variable into K8s Pod

Asked 2021-Dec-07 at 11:15

I would like to run a pod on one of my IoT devices. Each one of those devices contains an environment variable I want this pod to use. Is there any way to inject this env variable into the pod using build-in templating of helm/kubectl? I was trying the following on my deployment.yaml file:

1env: 
2  - name: XXX
3    value: $ENV_FROM_HOST
4

but when executing the pod and trying the get XXX value, I get the string $ENV_FROM_HOST instead of its value from the host:

1env: 
2  - name: XXX
3    value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6

Thanks.

ANSWER

Answered 2021-Dec-06 at 01:10

It's not possible to directly pass the host's env vars to the pods. I often do that by creating a ConfigMap.

  1. Create a ConfigMap with from-lireral option:

1env: 
2  - name: XXX
3    value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6kubectl create configmap testcm --from-literal=hostname=$HOSTNAME
7

  • Refer to that in the Pod's manifest:

    • 1env: 
2  - name: XXX
3    value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6kubectl create configmap testcm --from-literal=hostname=$HOSTNAME
7- name: TEST
8  valueFrom:
9    configMapKeyRef:
10      name: testcm
11      key: hostname
12

    This will inject the host's $HOSTNAME into the Pod's $TEST.

    If it's sensitive information, you can use Secrets instead of using ConfigMap.

    Source https://stackoverflow.com/questions/70237603

    QUESTION

    Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas

    Asked 2021-Nov-27 at 21:36

    I got this code to almost work, despite much ignorance. Please help on the home run!

    • Problem 1: INPUT:

    I have a long list of URLs (1000+) to read from and they are in a single column in .csv. I would prefer to read from that file than to paste them into code, like below.

    • Problem 2: OUTPUT:

    The source files actually have 3 drivers and 3 challenges each. In a separate python file, the below code finds, prints and saves all 3, but not when I'm using this dataframe below (see below - it only saves 2).

    • Problem 3: OUTPUT:

    I want the output (both files) to have URLs in column 0, and then drivers (or challenges) in the following columns. But what I've written here (probably the 'drop') makes them not only drop one row but also move across 2 columns.

    At the end I'm showing both the inputs and the current & desired output. Sorry for the long question. I'll be very grateful for any help!

    1import requests
2from bs4 import BeautifulSoup
3import pandas as pd
4
5urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
6dataframes = []
7dataframes2 = []
8
9for url in urls:
10    page = requests.get(url)
11    soup = BeautifulSoup(page.text, 'html.parser')
12    toc = soup.find("div", id="toc")
13
14    def get_drivers():
15        data = []
16        for x in toc.select('li:-soup-contains-own("Market drivers") li'):
17            data.append(x.get_text(strip=True))
18        df = pd.DataFrame(data, columns=[url])
19        dataframes.append(pd.DataFrame(df).drop(0, axis=0))
20        df2 = pd.concat(dataframes)
21        tdata = df2.T
22        tdata.to_csv(f'detail-dr.csv', header=True)
23
24    get_drivers()
25
26
27    def get_challenges():
28        data = []
29        for y in toc.select('li:-soup-contains-own("Market challenges") li'):
30            data.append(y.get_text(strip=True).replace('Table Impact of drivers and challenges', ''))
31        df = pd.DataFrame(data, columns=[url])
32        dataframes2.append(pd.DataFrame(df).drop(0, axis=0))
33        df2 = pd.concat(dataframes2)
34        tdata = df2.T
35        tdata.to_csv(f'detail-ch.csv', header=True)
36
37    get_challenges()
38

    The inputs look like this in each URL. They are just lists:

    Market drivers

    • Growing investment in fabs
    • Miniaturization of electronic products
    • Increasing demand for IoT devices

    Market challenges

    • Rapid technological changes in semiconductor industry
    • Volatility in semiconductor industry
    • Impact of technology chasm Table Impact of drivers and challenges

    My desired output for drivers is:

    0 1 2 3
    http/.../Global-Induction-Hobs-30196623/ Product innovations and new designs Increasing demand for convenient home appliances with changes in lifestyle patterns Growing adoption of energy-efficient appliances
    http/.../Global-Human-Capital-Management-30196628/ Demand for automated recruitment processes Increasing demand for unified solutions for all HR functions Increasing workforce diversity
    http/.../Global-Probe-Card-30196643/ Growing investment in fabs Miniaturization of electronic products Increasing demand for IoT devices

    But instead I get:

    0 1 2 3 4 5 6
    http/.../Global-Induction-Hobs-30196623/ Increasing demand for convenient home appliances with changes in lifestyle patterns Growing adoption of energy-efficient appliances
    http/.../Global-Human-Capital-Management-30196628/ Increasing demand for unified solutions for all HR functions Increasing workforce diversity
    http/.../Global-Probe-Card-30196643/ Miniaturization of electronic products Increasing demand for IoT devices

    ANSWER

    Answered 2021-Nov-27 at 21:36

    Store your data in a list of dicts, create a data frame from it. Split the list of drivers / challenges into single columns and concat it to the final data frame.

    Example 
    1import requests
2from bs4 import BeautifulSoup
3import pandas as pd
4
5urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
6dataframes = []
7dataframes2 = []
8
9for url in urls:
10    page = requests.get(url)
11    soup = BeautifulSoup(page.text, 'html.parser')
12    toc = soup.find("div", id="toc")
13
14    def get_drivers():
15        data = []
16        for x in toc.select('li:-soup-contains-own("Market drivers") li'):
17            data.append(x.get_text(strip=True))
18        df = pd.DataFrame(data, columns=[url])
19        dataframes.append(pd.DataFrame(df).drop(0, axis=0))
20        df2 = pd.concat(dataframes)
21        tdata = df2.T
22        tdata.to_csv(f'detail-dr.csv', header=True)
23
24    get_drivers()
25
26
27    def get_challenges():
28        data = []
29        for y in toc.select('li:-soup-contains-own("Market challenges") li'):
30            data.append(y.get_text(strip=True).replace('Table Impact of drivers and challenges', ''))
31        df = pd.DataFrame(data, columns=[url])
32        dataframes2.append(pd.DataFrame(df).drop(0, axis=0))
33        df2 = pd.concat(dataframes2)
34        tdata = df2.T
35        tdata.to_csv(f'detail-ch.csv', header=True)
36
37    get_challenges()
38import requests
39from bs4 import BeautifulSoup
40import pandas as pd
41
42urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
43data = []
44
45for url in urls:
46    page = requests.get(url)
47    soup = BeautifulSoup(page.text, 'html.parser')
48    toc = soup.find("div", id="toc")
49
50    def get_drivers():
51        data.append({
52            'url':url,
53            'type':'driver',
54            'list':[x.get_text(strip=True) for x in toc.select('li:-soup-contains-own("Market drivers") li')]
55        })
56
57    get_drivers()
58
59
60    def get_challenges():
61        data.append({
62            'url':url,
63            'type':'challenges',
64            'list':[x.text.replace('Table Impact of drivers and challenges','') for x in toc.select('li:-soup-contains-own("Market challenges") ul li') if x.text != 'Table Impact of drivers and challenges']
65        })
66
67    get_challenges()
68
69    
70pd.concat([pd.DataFrame(data)[['url','type']], pd.DataFrame(pd.DataFrame(data).list.tolist())],axis = 1)#.to_csv(sep='|')
71
    Output
    url type 0 1 2
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/ driver Product innovations and new designs Increasing demand for convenient home appliances with changes in lifestyle patterns Growing adoption of energy-efficient appliances
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/ challenges High cost limiting the adoption in the mass segment Health hazards related to induction hobs Limitation of using only flat - surface utensils and induction-specific cookware
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/ driver Demand for automated recruitment processes Increasing demand for unified solutions for all HR functions Increasing workforce diversity
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/ challenges Threat from open-source software High implementation and maintenance cost Threat to data security
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/ driver Growing investment in fabs Miniaturization of electronic products Increasing demand for IoT devices
    https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/ challenges Rapid technological changes in semiconductor industry Volatility in semiconductor industry Impact of technology chasm

    Source https://stackoverflow.com/questions/70128790

    QUESTION

    Generate all possible (public internet) IPV4 Address combinations in Python

    Asked 2021-Oct-26 at 08:43

    My goal is to loop through all possible ipv4 address that could be used in the web (like a computer, IoT device, etc, just not private networks) combinations in Python. As I was searching, I found a couple of solutions:

    1: Use a while loop and loop through the possible combinations (however, not sure this would stop, I ported this from a stackoverflow in C that I now cannot find):

    1i = 0
2while(i != -1):
3
4    b1 = (i >> 24) & 0xff
5    b2 = (i >> 16) & 0xff
6    b3 = (i >>  8) & 0xff
7    b4 = (i >> 4) & 0xff
8
9    i += 1
10

    2: Using a for loop: I found this code in a bash question: bash - Generate all possible ipv4 addresses using seq

    (c code):

    1i = 0
2while(i != -1):
3
4    b1 = (i >> 24) & 0xff
5    b2 = (i >> 16) & 0xff
6    b3 = (i >>  8) & 0xff
7    b4 = (i >> 4) & 0xff
8
9    i += 1
10int main() {
11  int h, i, j, k;
12
13  for (h = 0; h < 256; h++) {
14    for (i = 0; i < 256; i++) {
15      for (j = 0; j < 256; j++) {
16        for (k = 0; k < 256; k++) {
17          printf("%d.%d.%d.%d\n", h, i, j, k);
18        }
19      }
20    }
21  }
22  return 0;
23}
24

    I was wondering if there is a better way to do this? Both of these implementations allow 0.0.0.0 and private ips to be valid, so these are not solutions I want at the moment.

    ANSWER

    Answered 2021-Oct-26 at 06:50

    You can use ipaddress, it's in the standard library. is_private will exclude RFC1918 addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) as well as some others.

    1i = 0
2while(i != -1):
3
4    b1 = (i >> 24) & 0xff
5    b2 = (i >> 16) & 0xff
6    b3 = (i >>  8) & 0xff
7    b4 = (i >> 4) & 0xff
8
9    i += 1
10int main() {
11  int h, i, j, k;
12
13  for (h = 0; h < 256; h++) {
14    for (i = 0; i < 256; i++) {
15      for (j = 0; j < 256; j++) {
16        for (k = 0; k < 256; k++) {
17          printf("%d.%d.%d.%d\n", h, i, j, k);
18        }
19      }
20    }
21  }
22  return 0;
23}
24import ipaddress
25
26all_ipv4 = ipaddress.ip_network('0.0.0.0/0')
27
28for host in all_ipv4.hosts():
29    if host.is_private:
30        continue
31    print(host) #do your thing here
32

    Source https://stackoverflow.com/questions/69718005

    QUESTION

    Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres

    Asked 2021-Oct-25 at 12:39

    I am trying to set up a system to save historic data with the flow like this: Prosys OPC-UA Server Simulation -> OPC-UA FIWARE IoT Agent -> Orion Context Broker -> FIWARE Cygnus Connector -> PostgreSQL database.

    Here is the document I used to compose the docker-compose file:

    Historic-Context-Flume

    OPC-UA IoT Agent

    Here is the docker-compose and .env file I used

    docker-compose.yml

    1version: "3"
2#secrets:
3#   age_idm_auth:
4#      file: age_idm_auth.txt
5
6services:
7  iotage:
8    hostname: iotage
9    image: iotagent4fiware/iotagent-opcua:1.3.4
10    networks:
11      - hostnet
12      - iotnet
13    ports:
14      - "${AGENT_SERVER_PORT}:${AGENT_SERVER_PORT}"
15      - "4081:8080"
16    extra_hosts:
17      - "iotcarsrv:192.168.50.167"
18      - "HP:192.168.50.167"
19    depends_on:
20      - iotmongo
21      - orion
22    volumes:
23      - ./AGECONF:/opt/iotagent-opcua/conf
24      - ./certificates/charm:/opt/iotagent-opcua/certificates
25    command: /usr/bin/tail -f /var/log/lastlog
26
27  iotmongo:
28    hostname: iotmongo
29    image: mongo:3.4
30    networks:
31      - iotnet
32    volumes:
33      - iotmongo_data:/data/db
34      - iotmongo_conf:/data/configdb
35
36  ################ OCB ################
37
38  orion:
39    hostname: orion
40    image: fiware/orion:latest
41    networks:
42      - hostnet
43      - ocbnet
44    ports:
45      - "${ORION_PORT}:${ORION_PORT}"
46    depends_on:
47      - orion_mongo
48    #command: -dbhost mongo
49    entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
50
51  orion_mongo:
52    hostname: orion_mongo
53    image: mongo:3.4
54    networks:
55      ocbnet:
56        aliases:
57          - mongo
58    volumes:
59      - orion_mongo_data:/data/db
60      - orion_mongo_conf:/data/configdb
61    command: --nojournal
62
63  ############### CYGNUS ###############
64
65  cygnus:
66    image: fiware/cygnus-ngsi:${CYGNUS_VERSION}
67    hostname: cygnus
68    container_name: fiware-cygnus
69    networks:
70      - hostnet
71    depends_on:
72      - postgres-db
73    expose:
74      - "${CYGNUS_POSTGRESQL_SERVICE_PORT}" # 5055
75      - "${CYGNUS_API_PORT}" # 5080
76    ports:
77      - "${CYGNUS_POSTGRESQL_SERVICE_PORT}:${CYGNUS_POSTGRESQL_SERVICE_PORT}"
78      - "${CYGNUS_API_PORT}:${CYGNUS_API_PORT}"
79    environment:
80      - "CYGNUS_POSTGRESQL_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}"      
81      - "CYGNUS_POSTGRESQL_HOST=postgres-db" # Hostname of the PostgreSQL server used to persist historical contex
82      - "CYGNUS_POSTGRESQL_PORT=${POSTGRES_DB_PORT}" # Port that the PostgreSQL server uses to listen to commands
83      - "CYGNUS_POSTGRESQL_DATABASE=postgres"
84      - "CYGNUS_POSTGRESQL_USER=postgres" # Username for the PostgreSQL database user
85      - "CYGNUS_POSTGRESQL_PASS=password" # Password for the PostgreSQL database user
86      - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true" # Switch to enable caching within the PostgreSQL configuration
87      - "CYGNUS_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}" # Notification Port that Cygnus listens when subcr
88      - "CYGNUS_API_PORT=${CYGNUS_API_PORT}" # Port that Cygnus listens on for operational reasons
89      - "CYGNUS_LOG_LEVEL=DEBUG" # The logging level for Cygnus
90
91
92  postgres-db:
93    image: postgres
94    hostname: postgres-db
95    expose:
96      - "${POSTGRES_DB_PORT}"
97    ports:
98      - "${POSTGRES_DB_PORT}:${POSTGRES_DB_PORT}"
99    networks:
100      - hostnet
101    environment:
102      - "POSTGRES_PASSWORD=password"
103      - "POSTGRES_USER=postgres"
104      - "POSTGRES_DB=postgres"
105    volumes:
106      - postgres-db:/var/lib/postgresql/data
107
108volumes:
109  iotmongo_data:
110  iotmongo_conf:
111  orion_mongo_data:
112  orion_mongo_conf:
113  postgres-db:
114
115networks:
116  hostnet:
117  iotnet:
118  ocbnet:
119

    .env

    1version: "3"
2#secrets:
3#   age_idm_auth:
4#      file: age_idm_auth.txt
5
6services:
7  iotage:
8    hostname: iotage
9    image: iotagent4fiware/iotagent-opcua:1.3.4
10    networks:
11      - hostnet
12      - iotnet
13    ports:
14      - "${AGENT_SERVER_PORT}:${AGENT_SERVER_PORT}"
15      - "4081:8080"
16    extra_hosts:
17      - "iotcarsrv:192.168.50.167"
18      - "HP:192.168.50.167"
19    depends_on:
20      - iotmongo
21      - orion
22    volumes:
23      - ./AGECONF:/opt/iotagent-opcua/conf
24      - ./certificates/charm:/opt/iotagent-opcua/certificates
25    command: /usr/bin/tail -f /var/log/lastlog
26
27  iotmongo:
28    hostname: iotmongo
29    image: mongo:3.4
30    networks:
31      - iotnet
32    volumes:
33      - iotmongo_data:/data/db
34      - iotmongo_conf:/data/configdb
35
36  ################ OCB ################
37
38  orion:
39    hostname: orion
40    image: fiware/orion:latest
41    networks:
42      - hostnet
43      - ocbnet
44    ports:
45      - "${ORION_PORT}:${ORION_PORT}"
46    depends_on:
47      - orion_mongo
48    #command: -dbhost mongo
49    entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
50
51  orion_mongo:
52    hostname: orion_mongo
53    image: mongo:3.4
54    networks:
55      ocbnet:
56        aliases:
57          - mongo
58    volumes:
59      - orion_mongo_data:/data/db
60      - orion_mongo_conf:/data/configdb
61    command: --nojournal
62
63  ############### CYGNUS ###############
64
65  cygnus:
66    image: fiware/cygnus-ngsi:${CYGNUS_VERSION}
67    hostname: cygnus
68    container_name: fiware-cygnus
69    networks:
70      - hostnet
71    depends_on:
72      - postgres-db
73    expose:
74      - "${CYGNUS_POSTGRESQL_SERVICE_PORT}" # 5055
75      - "${CYGNUS_API_PORT}" # 5080
76    ports:
77      - "${CYGNUS_POSTGRESQL_SERVICE_PORT}:${CYGNUS_POSTGRESQL_SERVICE_PORT}"
78      - "${CYGNUS_API_PORT}:${CYGNUS_API_PORT}"
79    environment:
80      - "CYGNUS_POSTGRESQL_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}"      
81      - "CYGNUS_POSTGRESQL_HOST=postgres-db" # Hostname of the PostgreSQL server used to persist historical contex
82      - "CYGNUS_POSTGRESQL_PORT=${POSTGRES_DB_PORT}" # Port that the PostgreSQL server uses to listen to commands
83      - "CYGNUS_POSTGRESQL_DATABASE=postgres"
84      - "CYGNUS_POSTGRESQL_USER=postgres" # Username for the PostgreSQL database user
85      - "CYGNUS_POSTGRESQL_PASS=password" # Password for the PostgreSQL database user
86      - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true" # Switch to enable caching within the PostgreSQL configuration
87      - "CYGNUS_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}" # Notification Port that Cygnus listens when subcr
88      - "CYGNUS_API_PORT=${CYGNUS_API_PORT}" # Port that Cygnus listens on for operational reasons
89      - "CYGNUS_LOG_LEVEL=DEBUG" # The logging level for Cygnus
90
91
92  postgres-db:
93    image: postgres
94    hostname: postgres-db
95    expose:
96      - "${POSTGRES_DB_PORT}"
97    ports:
98      - "${POSTGRES_DB_PORT}:${POSTGRES_DB_PORT}"
99    networks:
100      - hostnet
101    environment:
102      - "POSTGRES_PASSWORD=password"
103      - "POSTGRES_USER=postgres"
104      - "POSTGRES_DB=postgres"
105    volumes:
106      - postgres-db:/var/lib/postgresql/data
107
108volumes:
109  iotmongo_data:
110  iotmongo_conf:
111  orion_mongo_data:
112  orion_mongo_conf:
113  postgres-db:
114
115networks:
116  hostnet:
117  iotnet:
118  ocbnet:
119# Orion
120ORION_PORT=1026
121
122# PostgreSQL
123POSTGRES_DB_PORT=5432
124
125# OPCUA IoT Agent
126AGENT_SERVER_PORT=4001
127
128# Cygnus
129CYGNUS_VERSION=2.10.0
130CYGNUS_API_PORT=5080
131CYGNUS_MYSQL_SERVICE_PORT=5050
132CYGNUS_MONGO_SERVICE_PORT=5051
133CYGNUS_CKAN_SERVICE_PORT=5052
134CYGNUS_HDFS_SERVICE_PORT=5053
135CYGNUS_CARTO_SERVICE_PORT=5054
136CYGNUS_POSTGRESQL_SERVICE_PORT=5055
137CYGNUS_ORION_SERVICE_PORT=5056
138CYGNUS_POSTGIS_SERVICE_PORT=5057
139CYGNUS_ELASTICSEARCH_SERVICE_PORT=5058
140CYGNUS_ARCGIS_SERVICE_PORT=5059
141

    I can see the system is up and no error informed from docker docker ps -a result

    However, when I use pgAdmin or docker exec to examine the postgres database there was nothing in it. \dt command of psql gave me Did not find any relation which means there was nothing logged into the database.

    Please provide help to combine these two components of FIWARE (the IoT OPCUA agent and the Cygnus). It was also not clear for me if the Cygnus connector would create the database for us.

    [EDIT] The component seems to be working: Simulation Server enter image description here

    All services enter image description here

    Cynus only infor me that 'Configuration has no change', when I used docker exec to get into the container, from Cygnus I was able to ping both orion, postgresql database containers enter image description here

    enter image description here

    In postgresql: I don't see the desired database or table enter image description here

    My system from the Simulation Server -> OPCUA IoT Agent -> Orion was working well because from docker logs I was still able to get the updated value of Simulation Server: enter image description here enter image description here

    ANSWER

    Answered 2021-Sep-01 at 10:31

    You have a complex scenario here, composed of an end-to-end chain of 5 components:

    • Prosys OPC-UA Server Simulation
    • OPC-UA FIWARE IoT Agent
    • Orion Context Broker
    • FIWARE Cygnus Connector
    • PostgreSQL database

    My recomendation here would be to check every step in the chain (looking in logs, etc.) to ensure everything is correct before of checking the next step.

    Source https://stackoverflow.com/questions/69002930

    QUESTION

    DRF testing - create object with a many-to-many relation

    Asked 2021-Oct-20 at 15:20

    I have the following model Project:

    1class Project(models.Model):
2    name = models.CharField(max_length=128)
3    slug = models.SlugField(blank=True)
4    assigned_to = models.ManyToManyField(
5        User, blank=True, related_name="assignees")
6    created_date = models.DateField(auto_now_add=True)
7    updated_date = models.DateField(auto_now_add=False, auto_now=True)
8

    If I need to create a new project, all I need to do is supply the name alone. This works for both the Admin dashboard and DRF APIVIEW. But when I try to test the functionality with DRF with an API call, I get the error: [b'{"assigned_to":["This field is required."]}'] Although the field is not required.

    My test code below

    1class Project(models.Model):
2    name = models.CharField(max_length=128)
3    slug = models.SlugField(blank=True)
4    assigned_to = models.ManyToManyField(
5        User, blank=True, related_name="assignees")
6    created_date = models.DateField(auto_now_add=True)
7    updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21    client = APIClient()
22    project = None
23
24    name = 'IOT on Blockchain'
25    dead_line = datetime.date(2021, 11, 21)
26    data = {
27        'name': name,
28        'dead_line': dead_line,
29    }
30
31    def create_user(self):
32        username = 'test_user1'
33        email = 'test.user1@gmail.com'
34        password = '@1234xyz@'
35        user_type = 'regular'
36        data = {'username': username,
37                'email': email,
38                'password': password,
39                'user_type': user_type,
40                }
41        return User.objects.create_user(**data)
42
43    def create_project(self):
44        project = Project.objects.create(**self.data)
45        user = self.create_user()
46        project.assigned_to.add(user)
47        return project
48
49    def test_create_project_without_api(self):
50        """
51        Ensure we can create a new project object.
52        """
53        self.project = self.create_project()
54        self.assertEqual(Project.objects.count(), 1)
55        self.assertEqual(self.project.name, 'IOT on Blockchain')
56        self.assertEqual(self.project.dead_line,
57                        datetime.date(2021, 11, 21))
58        self.assertFalse(self.project.reached_deadline)
59        self.assertEqual(self.project.days_to_deadline, 7)
60        # runs successfully
61
62    def test_create_project_with_api(self):
63        """
64        Ensure we can create a new project object with an
65        API call.
66        """
67        url = 'http://127.0.0.1:8000/api/projects'
68        project = self.client.post(url, self.data, format='json')
69        # project.data.assigned_to.set(self.create_user())
70        pprint(project.__dict__)
71        self.assertEqual(Project.objects.count(), 1)
72        self.assertEqual(self.project.name, 'IOT on Blockchain')
73        self.assertEqual(self.project.slug, 'iot-on-blockchain')
74        # does not run successfully (error mentioned in text body)
75
76    def test_delete_project(self):
77        """
78        We can delete a user
79        """
80        self.project = self.create_project()
81        self.project.delete()
82        self.assertEqual(Project.objects.count(), 0)
83

    Edit: Added serializer code

    1class Project(models.Model):
2    name = models.CharField(max_length=128)
3    slug = models.SlugField(blank=True)
4    assigned_to = models.ManyToManyField(
5        User, blank=True, related_name="assignees")
6    created_date = models.DateField(auto_now_add=True)
7    updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21    client = APIClient()
22    project = None
23
24    name = 'IOT on Blockchain'
25    dead_line = datetime.date(2021, 11, 21)
26    data = {
27        'name': name,
28        'dead_line': dead_line,
29    }
30
31    def create_user(self):
32        username = 'test_user1'
33        email = 'test.user1@gmail.com'
34        password = '@1234xyz@'
35        user_type = 'regular'
36        data = {'username': username,
37                'email': email,
38                'password': password,
39                'user_type': user_type,
40                }
41        return User.objects.create_user(**data)
42
43    def create_project(self):
44        project = Project.objects.create(**self.data)
45        user = self.create_user()
46        project.assigned_to.add(user)
47        return project
48
49    def test_create_project_without_api(self):
50        """
51        Ensure we can create a new project object.
52        """
53        self.project = self.create_project()
54        self.assertEqual(Project.objects.count(), 1)
55        self.assertEqual(self.project.name, 'IOT on Blockchain')
56        self.assertEqual(self.project.dead_line,
57                        datetime.date(2021, 11, 21))
58        self.assertFalse(self.project.reached_deadline)
59        self.assertEqual(self.project.days_to_deadline, 7)
60        # runs successfully
61
62    def test_create_project_with_api(self):
63        """
64        Ensure we can create a new project object with an
65        API call.
66        """
67        url = 'http://127.0.0.1:8000/api/projects'
68        project = self.client.post(url, self.data, format='json')
69        # project.data.assigned_to.set(self.create_user())
70        pprint(project.__dict__)
71        self.assertEqual(Project.objects.count(), 1)
72        self.assertEqual(self.project.name, 'IOT on Blockchain')
73        self.assertEqual(self.project.slug, 'iot-on-blockchain')
74        # does not run successfully (error mentioned in text body)
75
76    def test_delete_project(self):
77        """
78        We can delete a user
79        """
80        self.project = self.create_project()
81        self.project.delete()
82        self.assertEqual(Project.objects.count(), 0)
83class ProjectWriteSerializer(serializers.ModelSerializer):
84    """
85    This serializer is used for CREATE, UPDATE operations on the Project model.
86    """
87
88    # We receive list of user ids (ids[int] <= 0) by which we assign
89    # users to a project
90    assigned_to = serializers.PrimaryKeyRelatedField(
91        queryset=User.objects.all(), many=True)
92
93    class Meta:
94        model = Project
95        fields = ('id', 'name', 'slug', 'assigned_to')
96

    Any insights and help is very appreciated.

    ANSWER

    Answered 2021-Oct-20 at 15:20

    You have specified your own serializer field. As a result, it will no longer look at the blank=True part, and by default serializer fields are required. You can make these optional with:

    1class Project(models.Model):
2    name = models.CharField(max_length=128)
3    slug = models.SlugField(blank=True)
4    assigned_to = models.ManyToManyField(
5        User, blank=True, related_name="assignees")
6    created_date = models.DateField(auto_now_add=True)
7    updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21    client = APIClient()
22    project = None
23
24    name = 'IOT on Blockchain'
25    dead_line = datetime.date(2021, 11, 21)
26    data = {
27        'name': name,
28        'dead_line': dead_line,
29    }
30
31    def create_user(self):
32        username = 'test_user1'
33        email = 'test.user1@gmail.com'
34        password = '@1234xyz@'
35        user_type = 'regular'
36        data = {'username': username,
37                'email': email,
38                'password': password,
39                'user_type': user_type,
40                }
41        return User.objects.create_user(**data)
42
43    def create_project(self):
44        project = Project.objects.create(**self.data)
45        user = self.create_user()
46        project.assigned_to.add(user)
47        return project
48
49    def test_create_project_without_api(self):
50        """
51        Ensure we can create a new project object.
52        """
53        self.project = self.create_project()
54        self.assertEqual(Project.objects.count(), 1)
55        self.assertEqual(self.project.name, 'IOT on Blockchain')
56        self.assertEqual(self.project.dead_line,
57                        datetime.date(2021, 11, 21))
58        self.assertFalse(self.project.reached_deadline)
59        self.assertEqual(self.project.days_to_deadline, 7)
60        # runs successfully
61
62    def test_create_project_with_api(self):
63        """
64        Ensure we can create a new project object with an
65        API call.
66        """
67        url = 'http://127.0.0.1:8000/api/projects'
68        project = self.client.post(url, self.data, format='json')
69        # project.data.assigned_to.set(self.create_user())
70        pprint(project.__dict__)
71        self.assertEqual(Project.objects.count(), 1)
72        self.assertEqual(self.project.name, 'IOT on Blockchain')
73        self.assertEqual(self.project.slug, 'iot-on-blockchain')
74        # does not run successfully (error mentioned in text body)
75
76    def test_delete_project(self):
77        """
78        We can delete a user
79        """
80        self.project = self.create_project()
81        self.project.delete()
82        self.assertEqual(Project.objects.count(), 0)
83class ProjectWriteSerializer(serializers.ModelSerializer):
84    """
85    This serializer is used for CREATE, UPDATE operations on the Project model.
86    """
87
88    # We receive list of user ids (ids[int] <= 0) by which we assign
89    # users to a project
90    assigned_to = serializers.PrimaryKeyRelatedField(
91        queryset=User.objects.all(), many=True)
92
93    class Meta:
94        model = Project
95        fields = ('id', 'name', 'slug', 'assigned_to')
96class ProjectWriteSerializer(serializers.ModelSerializer):
97    assigned_to = serializers.PrimaryKeyRelatedField(
98        queryset=User.objects.all(),
99        many=True,
100        <strong>required=True</strong>
101    )
102
103    class Meta:
104        model = Project
105        fields = ('id', 'name', 'slug', 'assigned_to')

    Source https://stackoverflow.com/questions/69648631

    QUESTION

    Postgres March stange results

    Asked 2021-Oct-20 at 13:37

    I'm tring to calculate time prorata between two dates over multi month periods using SQL on PostgresSQL.

    My formula is pretty simple

    1select iot.*, 
2       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(  extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4  select DATE '2020-05-01' as DATE_DEBUT,   DATE '2020-05-31' as DATE_FIN
5) iot
6

    Give me expected result => 1

    But when I switch to march with

    1select iot.*, 
2       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(  extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4  select DATE '2020-05-01' as DATE_DEBUT,   DATE '2020-05-31' as DATE_FIN
5) iot
6select iot.*,   
7       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
8from (
9   select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
10) iot
11

    I've got a weird result of 1.0344827586206897

    Any ideas?

    ANSWER

    Answered 2021-Oct-20 at 13:37

    This is caused by date_trunc('month', this function returns a date with a timezone, but your whole calculus doesn't handle the timezone.

    If you force Postgres to ignore the timezone, it should work:

    1select iot.*, 
2       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(  extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4  select DATE '2020-05-01' as DATE_DEBUT,   DATE '2020-05-31' as DATE_FIN
5) iot
6select iot.*,   
7       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
8from (
9   select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
10) iot
11select iot.*,   
12       (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)::timestamp without time zone), 0::double precision) as prorata
13from (
14   select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
15) iot
16

    I just cast date_trunc with ::timestamp without time zone.

    Source https://stackoverflow.com/questions/69646843

    Community Discussions contain sources that include Stack Exchange Network

    Tutorials and Learning Resources in IOT

    Tutorials and Learning Resources are not available at this moment for IOT

    Share this Page

    share link

    Get latest updates on IOT

    Open Weaver – Develop Applications Faster with Open Source

    kandi

    Community and Support

    Company

    Follow

    • © 2023 Open Weaver Inc.