Popular New Releases in IOT
No Popular Releases are available at this moment for IOT
Popular Libraries in IOT
No Trending Libraries are available at this moment for IOT
Trending New libraries in IOT
No Trending Libraries are available at this moment for IOT
Top Authors in IOT
No Top Authors are available at this moment for IOT.
Trending Kits in IOT
No Trending Kits are available at this moment for IOT
Trending Discussions on IOT
Azure IoT Central architecture - how are Digital Twins implemented and managed?
Cloud2Edge package: error connectivity:connection.id.enforcement.failed
Connecting Waveshare LCD1602 RGB to Raspberry using C#
Why does the error "Unable to execute HTTP request" occur after unregistering ConnectivityManager.NetworkCallback?
Inject host's environment variable into K8s Pod
Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas
Generate all possible (public internet) IPV4 Address combinations in Python
Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres
DRF testing - create object with a many-to-many relation
Postgres March stange results
QUESTION
Azure IoT Central architecture - how are Digital Twins implemented and managed?
Asked 2022-Mar-17 at 01:54I have a system with IoT Hub to ingests events from devices and Device Provisioning Service to provision devices. IoT Hub C# SDKs are used for the management of device tags and desired properties (IoT Hub device twins), and to invoke direct methods or schedule jobs.
Recently I've been experimenting with Azure IoT Central. While I don't plan to use it, I've found Digital Twins (that are being used on Azure IoT Central) to offer a very good way of managing IoT devices and I would like to emulate the same kind of functionality and capabilities on my IoT system.
The high-level architecture of IoT Central does not seem to indicate the services or logic used to manage Digital Twins.
As far as I understand, there are two ways you can start using Digital Twins:
- Plug and Play Digital Twins
- Azure Digital Twins service
Question - is Azure IoT Central purely based on Plug and Play Digital Twins and/or does it also use Azure Digital Twins service?
ANSWER
Answered 2022-Mar-02 at 08:02Yes, IOT Central is purely based on plug and play Digital Twins
plug and play Digital Twins enables solution builders to integrate IoT devices with their solutions without any manual configuration.
Azure Digital Twins can be used to design a digital twin architecture that represents actual IoT devices in a wider cloud solution, and which connects to IoT Hub device twins to send and receive live data.
Reference link: https://docs.microsoft.com/en-us/azure/iot-develop/overview-iot-plug-and-play https://docs.microsoft.com/en-us/azure/digital-twins/overview#:~:text=What%20is%20Azure%20Digital%20Twins%3F%201%20Azure%20Digital,solution.%203%20Service%20limits.%20...%204%20Terminology.
QUESTION
Cloud2Edge package: error connectivity:connection.id.enforcement.failed
Asked 2022-Feb-14 at 07:56I'm testing a deployment of the Eclipse IoT Cloud2Edge package and have followed the instructions here https://www.eclipse.org/packages/packages/cloud2edge/tour/ to test. After creating the new tenant and device, and configuring the connection between Hono and Ditto, I can send telemetry to the new device via the Hono http adapter as shown here:
1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2 "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3 "headers": {},
4 "path": "/features/temperature/properties/value",
5 "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
10
and expected to see this property value updated in Ditto. The updated device property value does not update in Ditto, and when I check the Ditto logs I see the following entries:
1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2 "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3 "headers": {},
4 "path": "/features/temperature/properties/value",
5 "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
102022-02-13 20:11:35,265 INFO [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Received message from AMQP 1.0 with externalMessageHeaders: {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
112022-02-13 20:11:35,271 INFO [81c41f10-4d59-435b-8ae1-bf5194dcf6bf] o.e.d.c.s.m.InboundDispatchingSink - onMapped mappedHeaders ImmutableDittoHeaders [{ditto-entity-id=thing:my-tenant:org.acme:my-device-1, ditto-inbound-payload-mapper=default, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=81c41f10-4d59-435b-8ae1-bf5194dcf6bf}]
122022-02-13 20:11:35,278 INFO [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink - Got exception <connectivity:connection.id.enforcement.failed> when processing external message with mapper <default>: <The configured filters could not be matched against the given target with ID 'org.acme:my-device-1'.>
132022-02-13 20:11:35,278 INFO [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink - Resolved mapped headers of ImmutableDittoHeaders [{ditto-inbound-payload-mapper=default, ditto-entity-id=thing:my-tenant:org.acme:my-device-1, response-required=false, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=b3b11410-6df8-4bfc-a940-fafa87d65be2}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={hono-device-id={{ header:device_id }}, content-type={{ header:content-type }}}]] : and external headers {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
142022-02-13 20:11:35,283 INFO [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Acking <ID:AMQP_NO_PREFIX:GenericSenderLink-12> with original external message headers=<{orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}>, isSuccess=<true>, ackType=<1 accepted>
15
I think the problem is the "connectivity:connection.id.enforcement.failed" error but I don't know how to troubleshoot. Any advice appreciated.
ANSWER
Answered 2022-Feb-14 at 07:56What you configured is the Connection source enforcement which makes sure that a Hono device (identified via the AMQP header device_id
) may only updates the twin with the same "thing id" in Ditto.
That enforcement fails as your thingId you set in the Ditto Protocol JSON is my-tenant:org.acme:my-device-1
- the topic
's first segment is the namespace, the second segment the name - combined those 2 segments become the "thing ID", see also: Protocol topic specification.
So you probably want to send the following message instead:
1curl -i -u my-auth-id-1@my-tenant:my-password -H 'application/json' --data-binary '{
2 "topic": "my-tenant/org.acme:my-device-1/things/twin/commands/modify",
3 "headers": {},
4 "path": "/features/temperature/properties/value",
5 "value": 53
6}' http://${HTTP_ADAPTER_IP}:${HTTP_ADAPTER_PORT_HTTP}/telemetry
7HTTP/1.1 202 Accepted
8vary: origin
9content-length: 0
102022-02-13 20:11:35,265 INFO [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Received message from AMQP 1.0 with externalMessageHeaders: {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
112022-02-13 20:11:35,271 INFO [81c41f10-4d59-435b-8ae1-bf5194dcf6bf] o.e.d.c.s.m.InboundDispatchingSink - onMapped mappedHeaders ImmutableDittoHeaders [{ditto-entity-id=thing:my-tenant:org.acme:my-device-1, ditto-inbound-payload-mapper=default, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=81c41f10-4d59-435b-8ae1-bf5194dcf6bf}]
122022-02-13 20:11:35,278 INFO [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink - Got exception <connectivity:connection.id.enforcement.failed> when processing external message with mapper <default>: <The configured filters could not be matched against the given target with ID 'org.acme:my-device-1'.>
132022-02-13 20:11:35,278 INFO [b3b11410-6df8-4bfc-a940-fafa87d65be2] o.e.d.c.s.m.InboundDispatchingSink - Resolved mapped headers of ImmutableDittoHeaders [{ditto-inbound-payload-mapper=default, ditto-entity-id=thing:my-tenant:org.acme:my-device-1, response-required=false, content-type=application/x-www-form-urlencoded, hono-device-id=org.acme:my-device-1, ditto-reply-target=0, ditto-expected-response-types=["response","error"], ditto-origin=hono-connection-for-my-tenant, ditto-auth-context={"type":"pre-authenticated-connection","subjects":["pre-authenticated:hono-connection"]}, correlation-id=b3b11410-6df8-4bfc-a940-fafa87d65be2}] : with HeaderMapping Optional[ImmutableHeaderMapping [mapping={hono-device-id={{ header:device_id }}, content-type={{ header:content-type }}}]] : and external headers {orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}
142022-02-13 20:11:35,283 INFO [] o.e.d.c.s.m.a.AmqpConsumerActor akka://ditto-cluster/system/sharding/connection/3/hono-connection-for-my-tenant/pa/$a/c1/amqpConsumerActor-0-telemetry%2Fmy-tenant-010 - Acking <ID:AMQP_NO_PREFIX:GenericSenderLink-12> with original external message headers=<{orig_adapter=hono-http, qos=0, device_id=org.acme:my-device-1, creation-time=1644783095260, message-id=ID:AMQP_NO_PREFIX:GenericSenderLink-12, content-type=application/x-www-form-urlencoded, to=telemetry/my-tenant, orig_address=/telemetry}>, isSuccess=<true>, ackType=<1 accepted>
15{
16 "topic": "org.acme/my-device-1/things/twin/commands/modify",
17...
18}
19
QUESTION
Connecting Waveshare LCD1602 RGB to Raspberry using C#
Asked 2022-Jan-01 at 18:49I am trying to connect LCD1602 RBG Waveshare to the Raspberry using C#. I connected it to the Raspberry and set the permissions, now trying to pass some data. The code below run all lines, but the LCD is not reacting. If anyone can advise me.
1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11 public class Program
12 {
13 static void Main(string[] args)
14 {
15 Console.WriteLine("Test1");
16
17 using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18 //BCM2835
19 using var driver = new Pcf8574(i2c);
20 using var lcd = new Lcd1602(registerSelectPin: 0,
21 enablePin: 2,
22 dataPins: new int[] { 4, 5, 6, 7 },
23 backlightPin: 3,
24 backlightBrightness: 0.3f,
25 readWritePin: 1,
26 controller: new GpioController(PinNumberingScheme.Logical, driver));
27 int counter = 0;
28
29 while (counter <= 4)
30 {
31 lcd.Clear();
32
33 lcd.SetCursorPosition(0, 0);
34 lcd.Write("TestText1");
35
36 lcd.SetCursorPosition(0, 1);
37 lcd.Write("TestText2");
38
39 Thread.Sleep(2000);
40 counter++;
41 }
42 }
43 }
44}
45
1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11 public class Program
12 {
13 static void Main(string[] args)
14 {
15 Console.WriteLine("Test1");
16
17 using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18 //BCM2835
19 using var driver = new Pcf8574(i2c);
20 using var lcd = new Lcd1602(registerSelectPin: 0,
21 enablePin: 2,
22 dataPins: new int[] { 4, 5, 6, 7 },
23 backlightPin: 3,
24 backlightBrightness: 0.3f,
25 readWritePin: 1,
26 controller: new GpioController(PinNumberingScheme.Logical, driver));
27 int counter = 0;
28
29 while (counter <= 4)
30 {
31 lcd.Clear();
32
33 lcd.SetCursorPosition(0, 0);
34 lcd.Write("TestText1");
35
36 lcd.SetCursorPosition(0, 1);
37 lcd.Write("TestText2");
38
39 Thread.Sleep(2000);
40 counter++;
41 }
42 }
43 }
44}
45 0 1 2 3 4 5 6 7 8 9 a b c d e f
4600: -- -- -- -- -- -- -- --
4710: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4820: 20 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4930: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- 3e --
5040: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5150: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5260: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5370: 70 -- -- -- -- -- -- --
54
ANSWER
Answered 2022-Jan-01 at 18:49The controller for these displays is normally a HD44780 or compatible. If the python sample works fine, then this should do:
1using System;
2using System.Device.Gpio;
3using System.Threading.Tasks;
4using System.Device.I2c;
5using Iot.Device.CharacterLcd;
6using Iot.Device.Pcx857x;
7using System.Threading;
8
9namespace pi_project
10{
11 public class Program
12 {
13 static void Main(string[] args)
14 {
15 Console.WriteLine("Test1");
16
17 using I2cDevice i2c = I2cDevice.Create(new I2cConnectionSettings(1, 0x3e));
18 //BCM2835
19 using var driver = new Pcf8574(i2c);
20 using var lcd = new Lcd1602(registerSelectPin: 0,
21 enablePin: 2,
22 dataPins: new int[] { 4, 5, 6, 7 },
23 backlightPin: 3,
24 backlightBrightness: 0.3f,
25 readWritePin: 1,
26 controller: new GpioController(PinNumberingScheme.Logical, driver));
27 int counter = 0;
28
29 while (counter <= 4)
30 {
31 lcd.Clear();
32
33 lcd.SetCursorPosition(0, 0);
34 lcd.Write("TestText1");
35
36 lcd.SetCursorPosition(0, 1);
37 lcd.Write("TestText2");
38
39 Thread.Sleep(2000);
40 counter++;
41 }
42 }
43 }
44}
45 0 1 2 3 4 5 6 7 8 9 a b c d e f
4600: -- -- -- -- -- -- -- --
4710: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4820: 20 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
4930: -- -- -- -- -- -- -- -- -- -- -- -- 3c -- 3e --
5040: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5150: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5260: 60 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
5370: 70 -- -- -- -- -- -- --
54var i2cLcdDevice = I2cDevice.Create(new I2cConnectionSettings(busId: 1, deviceAddress: 0x3E));
55var i2cRgbDevice = I2cDevice.Create(new I2cConnectionSettings(busId: 1, deviceAddress: 0x60));
56using LcdRgb lcd = new LcdRgb(new Size(16, 2), i2cLcdDevice, i2cRgbDevice);
57{
58 lcd.Write("Hello World!");
59 lcd.SetBacklightColor(Color.Azure);
60}
61
(for reasons beyond me, the python sample defines the addresses as 0x7c >> 1
and 0xc0 >> 1
, which are 0x3E and 0x60, which aligns with your device scan)
QUESTION
Why does the error "Unable to execute HTTP request" occur after unregistering ConnectivityManager.NetworkCallback?
Asked 2021-Dec-07 at 17:48In an Android app which uses AWS services, if I deregister a registered ConnectivityManager.NetworkCallback
, the app can no longer contact AWS services. I am uncertain why this is occurring, or how to contact the AWS services again. Currently, the only way to reconnect to AWS is to terminate the app and restart it.
To elaborate, one function of the app is to connect the user's Android device to a different WiFi network hotspot for the purposes of setting up an IoT device. At this moment of use, the user would be logged in using AWS's Cognito service. Because how an app can connect to WiFi was changed starting with API level 29, this associated code is only invoked on such devices and the problem is isolated to said devices. Here is the relevant snippet for how the connection is being created:
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18
Unregistering the callback occurs if the setup is completed, if an error is encountered, or if the user decides to navigate backwards out of this setup. Regardless, this is the code used to accomplish unregistering:
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19 .getApplication()
20 .getApplicationContext()
21 .getSystemService(Context.CONNECTIVITY_SERVICE);
22 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23 if (connectivityManager != null) {
24 try {
25 connectivityManager.unregisterNetworkCallback(networkCallback);
26 } catch (IllegalArgumentException e) {
27 // Network is already unregistered.
28 e.printStackTrace();
29 }
30 }
31
This code successfully invokes in all circumstances. At this point, the user is no longer connected to the WiFi hotspot. In my testing, I would now be connected back to my home WiFi with my mobile network available too. However, the app can no longer contact AWS. Considering that my device is still connected to a WiFi network AND my mobile network, I would not expect this outcome. Here is an example stack trace for trying to contact Cognito (although the same sort of error occurs if DynamoDB is contacted, for example):
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19 .getApplication()
20 .getApplicationContext()
21 .getSystemService(Context.CONNECTIVITY_SERVICE);
22 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23 if (connectivityManager != null) {
24 try {
25 connectivityManager.unregisterNetworkCallback(networkCallback);
26 } catch (IllegalArgumentException e) {
27 // Network is already unregistered.
28 e.printStackTrace();
29 }
30 }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err: at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err: at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err: at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err: at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err: at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err: at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err: at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err: at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err: at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err: at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err: at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err: at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err: at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err: at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err: at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err: ... 7 more
62
I have isolated it to specifically when the app performs .unregisterNetworkCallback()
to be the cause. If this is not invoked, then the app will not lose the ability to contact AWS.
Additionally, when attempting any AWS connection after unregistering, many of these statements will print into the log before the exception above is logged:
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19 .getApplication()
20 .getApplicationContext()
21 .getSystemService(Context.CONNECTIVITY_SERVICE);
22 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23 if (connectivityManager != null) {
24 try {
25 connectivityManager.unregisterNetworkCallback(networkCallback);
26 } catch (IllegalArgumentException e) {
27 // Network is already unregistered.
28 e.printStackTrace();
29 }
30 }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err: at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err: at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err: at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err: at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err: at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err: at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err: at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err: at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err: at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err: at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err: at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err: at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err: at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err: at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err: at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err: ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63
In my research, it seems to have something to do with Samsung devices. I am testing this on a Samsung Galaxy S10 Lite. I am not sure if this is related to the issue at hand.
Why is this occurring? Am I somehow using the ConnectivityManager.NetworkCallback
incorrectly, either in registering or unregistering it? Is it an issue with AWS?
ANSWER
Answered 2021-Dec-07 at 17:48I realized the issue involved a snippet of code outside of the code shared in this question. The NetworkCallback's onAvailable()
is performed as such:
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19 .getApplication()
20 .getApplicationContext()
21 .getSystemService(Context.CONNECTIVITY_SERVICE);
22 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23 if (connectivityManager != null) {
24 try {
25 connectivityManager.unregisterNetworkCallback(networkCallback);
26 } catch (IllegalArgumentException e) {
27 // Network is already unregistered.
28 e.printStackTrace();
29 }
30 }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err: at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err: at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err: at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err: at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err: at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err: at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err: at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err: at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err: at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err: at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err: at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err: at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err: at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err: at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err: at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err: ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63 @Override
64 public void onAvailable(@NonNull Network network) {
65 connectivityManager.bindProcessToNetwork(network);
66 }
67
The reason we are calling .bindProcessToNetwork()
is so that the app can still access the internet beyond just the WiFi network.
This is key to why the app is losing connection to AWS after calling .unregisterNetworkCallback()
; it is being unregistered but it is still bound to the network we just unregistered. Therefore, when one goes to unregister the NetworkCallback, be sure to first invoke .bindProcessToNetwork()
and pass null to clear out the binding:
1 NetworkRequest request =
2 new NetworkRequest.Builder()
3 .addTransportType(TRANSPORT_WIFI)
4 .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
5 .setNetworkSpecifier(specifier)
6 .build();
7
8 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
9 .getApplication()
10 .getSystemService(Context.CONNECTIVITY_SERVICE);
11
12 ConnectivityManager.NetworkCallback networkCallback = viewModel.getNetworkCallback();
13
14 if (connectivityManager != null && request != null) {
15 // Request the network.
16 connectivityManager.requestNetwork(request, networkCallback, TIMEOUT_AMOUNT);
17 }
18 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
19 .getApplication()
20 .getApplicationContext()
21 .getSystemService(Context.CONNECTIVITY_SERVICE);
22 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
23 if (connectivityManager != null) {
24 try {
25 connectivityManager.unregisterNetworkCallback(networkCallback);
26 } catch (IllegalArgumentException e) {
27 // Network is already unregistered.
28 e.printStackTrace();
29 }
30 }
31W/System.err: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
32W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:456)
33W/System.err: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:229)
34W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.invoke(AmazonCognitoIdentityProviderClient.java:6329)
35W/System.err: at com.amazonaws.services.cognitoidentityprovider.AmazonCognitoIdentityProviderClient.getUser(AmazonCognitoIdentityProviderClient.java:4052)
36W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.getUserDetailsInternal(CognitoUser.java:1487)
37W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser.access$700(CognitoUser.java:133)
38W/System.err: at com.amazonaws.mobileconnectors.cognitoidentityprovider.CognitoUser$10.run(CognitoUser.java:1433)
39W/System.err: at java.lang.Thread.run(Thread.java:923)
40W/System.err: Caused by: java.net.UnknownHostException: Unable to resolve host "cognito-idp.us-east-2.amazonaws.com": No address associated with hostname
41W/System.err: at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:124)
42W/System.err: at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:103)
43W/System.err: at java.net.InetAddress.getAllByName(InetAddress.java:1152)
44W/System.err: at com.android.okhttp.Dns$1.lookup(Dns.java:41)
45W/System.err: at com.android.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:178)
46W/System.err: at com.android.okhttp.internal.http.RouteSelector.nextProxy(RouteSelector.java:144)
47W/System.err: at com.android.okhttp.internal.http.RouteSelector.next(RouteSelector.java:86)
48W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:192)
49W/System.err: at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:144)
50W/System.err: at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:106)
51W/System.err: at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:400)
52W/System.err: at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:333)
53W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:465)
54W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:131)
55W/System.err: at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:262)
56W/System.err: at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:219)
57W/System.err: at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:30)
58W/System.err: at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
59W/System.err: at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
60W/System.err: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:386)
61W/System.err: ... 7 more
62I/System.out: (HTTPLog)-Static: isSBSettingEnabled false
63 @Override
64 public void onAvailable(@NonNull Network network) {
65 connectivityManager.bindProcessToNetwork(network);
66 }
67 ConnectivityManager connectivityManager = (ConnectivityManager) viewModel
68 .getApplication()
69 .getApplicationContext()
70 .getSystemService(Context.CONNECTIVITY_SERVICE);
71 WiFiNetworkCallback networkCallback = viewModel.getNetworkCallback();
72 if (connectivityManager != null) {
73 try {
74 // Remove the app from the network callback first before deregistering it. Otherwise,
75 // the app will be stuck on the deregistered network (giving it no connectivity).
76 connectivityManager.bindProcessToNetwork(null);
77 connectivityManager.unregisterNetworkCallback(networkCallback);
78 } catch (IllegalArgumentException e) {
79 // Network is already unregistered.
80 e.printStackTrace();
81 }
82 }
83
I came to this answer by reading this answer on a different question. It gives a great overview of how the whole WiFi connection/management process works for Android API 29 and higher, including this final part of disconnecting and allowing the user to continue with the app experience on their normal WiFi or cellular network.
QUESTION
Inject host's environment variable into K8s Pod
Asked 2021-Dec-07 at 11:15I would like to run a pod on one of my IoT devices.
Each one of those devices contains an environment variable I want this pod to use.
Is there any way to inject this env variable into the pod using build-in templating of helm
/kubectl
?
I was trying the following on my deployment.yaml
file:
1env:
2 - name: XXX
3 value: $ENV_FROM_HOST
4
but when executing the pod and trying the get XXX
value, I get the string $ENV_FROM_HOST
instead of its value from the host:
1env:
2 - name: XXX
3 value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6
Thanks.
ANSWER
Answered 2021-Dec-06 at 01:10It's not possible to directly pass the host's env vars to the pods. I often do that by creating a ConfigMap.
Create a ConfigMap with
from-lireral
option:
1env:
2 - name: XXX
3 value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6kubectl create configmap testcm --from-literal=hostname=$HOSTNAME
7
Refer to that in the Pod's manifest:
1env:
2 - name: XXX
3 value: $ENV_FROM_HOST
4$ echo $XXX
5$ENV_FROM_HOST
6kubectl create configmap testcm --from-literal=hostname=$HOSTNAME
7- name: TEST
8 valueFrom:
9 configMapKeyRef:
10 name: testcm
11 key: hostname
12
This will inject the host's $HOSTNAME into the Pod's $TEST.
If it's sensitive information, you can use Secrets instead of using ConfigMap.
QUESTION
Reading URLs from .csv and appending scrape results below previous with Python, BeautifulSoup, Pandas
Asked 2021-Nov-27 at 21:36I got this code to almost work, despite much ignorance. Please help on the home run!
- Problem 1: INPUT:
I have a long list of URLs (1000+) to read from and they are in a single column in .csv. I would prefer to read from that file than to paste them into code, like below.
- Problem 2: OUTPUT:
The source files actually have 3 drivers and 3 challenges each. In a separate python file, the below code finds, prints and saves all 3, but not when I'm using this dataframe below (see below - it only saves 2).
- Problem 3: OUTPUT:
I want the output (both files) to have URLs in column 0, and then drivers (or challenges) in the following columns. But what I've written here (probably the 'drop') makes them not only drop one row but also move across 2 columns.
At the end I'm showing both the inputs and the current & desired output. Sorry for the long question. I'll be very grateful for any help!
1import requests
2from bs4 import BeautifulSoup
3import pandas as pd
4
5urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
6dataframes = []
7dataframes2 = []
8
9for url in urls:
10 page = requests.get(url)
11 soup = BeautifulSoup(page.text, 'html.parser')
12 toc = soup.find("div", id="toc")
13
14 def get_drivers():
15 data = []
16 for x in toc.select('li:-soup-contains-own("Market drivers") li'):
17 data.append(x.get_text(strip=True))
18 df = pd.DataFrame(data, columns=[url])
19 dataframes.append(pd.DataFrame(df).drop(0, axis=0))
20 df2 = pd.concat(dataframes)
21 tdata = df2.T
22 tdata.to_csv(f'detail-dr.csv', header=True)
23
24 get_drivers()
25
26
27 def get_challenges():
28 data = []
29 for y in toc.select('li:-soup-contains-own("Market challenges") li'):
30 data.append(y.get_text(strip=True).replace('Table Impact of drivers and challenges', ''))
31 df = pd.DataFrame(data, columns=[url])
32 dataframes2.append(pd.DataFrame(df).drop(0, axis=0))
33 df2 = pd.concat(dataframes2)
34 tdata = df2.T
35 tdata.to_csv(f'detail-ch.csv', header=True)
36
37 get_challenges()
38
The inputs look like this in each URL. They are just lists:
Market drivers
- Growing investment in fabs
- Miniaturization of electronic products
- Increasing demand for IoT devices
Market challenges
- Rapid technological changes in semiconductor industry
- Volatility in semiconductor industry
- Impact of technology chasm Table Impact of drivers and challenges
My desired output for drivers is:
0 | 1 | 2 | 3 |
---|---|---|---|
http/.../Global-Induction-Hobs-30196623/ | Product innovations and new designs | Increasing demand for convenient home appliances with changes in lifestyle patterns | Growing adoption of energy-efficient appliances |
http/.../Global-Human-Capital-Management-30196628/ | Demand for automated recruitment processes | Increasing demand for unified solutions for all HR functions | Increasing workforce diversity |
http/.../Global-Probe-Card-30196643/ | Growing investment in fabs | Miniaturization of electronic products | Increasing demand for IoT devices |
But instead I get:
0 | 1 | 2 | 3 | 4 | 5 | 6 |
---|---|---|---|---|---|---|
http/.../Global-Induction-Hobs-30196623/ | Increasing demand for convenient home appliances with changes in lifestyle patterns | Growing adoption of energy-efficient appliances | ||||
http/.../Global-Human-Capital-Management-30196628/ | Increasing demand for unified solutions for all HR functions | Increasing workforce diversity | ||||
http/.../Global-Probe-Card-30196643/ | Miniaturization of electronic products | Increasing demand for IoT devices |
ANSWER
Answered 2021-Nov-27 at 21:36Store your data in a list of dicts, create a data frame from it. Split the list of drivers
/ challenges
into single columns
and concat it to the final data frame.
1import requests
2from bs4 import BeautifulSoup
3import pandas as pd
4
5urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
6dataframes = []
7dataframes2 = []
8
9for url in urls:
10 page = requests.get(url)
11 soup = BeautifulSoup(page.text, 'html.parser')
12 toc = soup.find("div", id="toc")
13
14 def get_drivers():
15 data = []
16 for x in toc.select('li:-soup-contains-own("Market drivers") li'):
17 data.append(x.get_text(strip=True))
18 df = pd.DataFrame(data, columns=[url])
19 dataframes.append(pd.DataFrame(df).drop(0, axis=0))
20 df2 = pd.concat(dataframes)
21 tdata = df2.T
22 tdata.to_csv(f'detail-dr.csv', header=True)
23
24 get_drivers()
25
26
27 def get_challenges():
28 data = []
29 for y in toc.select('li:-soup-contains-own("Market challenges") li'):
30 data.append(y.get_text(strip=True).replace('Table Impact of drivers and challenges', ''))
31 df = pd.DataFrame(data, columns=[url])
32 dataframes2.append(pd.DataFrame(df).drop(0, axis=0))
33 df2 = pd.concat(dataframes2)
34 tdata = df2.T
35 tdata.to_csv(f'detail-ch.csv', header=True)
36
37 get_challenges()
38import requests
39from bs4 import BeautifulSoup
40import pandas as pd
41
42urls = ['https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/', 'https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/']
43data = []
44
45for url in urls:
46 page = requests.get(url)
47 soup = BeautifulSoup(page.text, 'html.parser')
48 toc = soup.find("div", id="toc")
49
50 def get_drivers():
51 data.append({
52 'url':url,
53 'type':'driver',
54 'list':[x.get_text(strip=True) for x in toc.select('li:-soup-contains-own("Market drivers") li')]
55 })
56
57 get_drivers()
58
59
60 def get_challenges():
61 data.append({
62 'url':url,
63 'type':'challenges',
64 'list':[x.text.replace('Table Impact of drivers and challenges','') for x in toc.select('li:-soup-contains-own("Market challenges") ul li') if x.text != 'Table Impact of drivers and challenges']
65 })
66
67 get_challenges()
68
69
70pd.concat([pd.DataFrame(data)[['url','type']], pd.DataFrame(pd.DataFrame(data).list.tolist())],axis = 1)#.to_csv(sep='|')
71
url | type | 0 | 1 | 2 |
---|---|---|---|---|
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/ | driver | Product innovations and new designs | Increasing demand for convenient home appliances with changes in lifestyle patterns | Growing adoption of energy-efficient appliances |
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Induction-Hobs-30196623/ | challenges | High cost limiting the adoption in the mass segment | Health hazards related to induction hobs | Limitation of using only flat - surface utensils and induction-specific cookware |
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/ | driver | Demand for automated recruitment processes | Increasing demand for unified solutions for all HR functions | Increasing workforce diversity |
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Human-Capital-Management-30196628/ | challenges | Threat from open-source software | High implementation and maintenance cost | Threat to data security |
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/ | driver | Growing investment in fabs | Miniaturization of electronic products | Increasing demand for IoT devices |
https://www.marketresearch.com/Infiniti-Research-Limited-v2680/Global-Probe-Card-30196643/ | challenges | Rapid technological changes in semiconductor industry | Volatility in semiconductor industry | Impact of technology chasm |
QUESTION
Generate all possible (public internet) IPV4 Address combinations in Python
Asked 2021-Oct-26 at 08:43My goal is to loop through all possible ipv4 address that could be used in the web (like a computer, IoT device, etc, just not private networks) combinations in Python. As I was searching, I found a couple of solutions:
1: Use a while loop and loop through the possible combinations (however, not sure this would stop, I ported this from a stackoverflow in C that I now cannot find):
1i = 0
2while(i != -1):
3
4 b1 = (i >> 24) & 0xff
5 b2 = (i >> 16) & 0xff
6 b3 = (i >> 8) & 0xff
7 b4 = (i >> 4) & 0xff
8
9 i += 1
10
2:
Using a for
loop:
I found this code in a bash question: bash - Generate all possible ipv4 addresses using seq
(c code):
1i = 0
2while(i != -1):
3
4 b1 = (i >> 24) & 0xff
5 b2 = (i >> 16) & 0xff
6 b3 = (i >> 8) & 0xff
7 b4 = (i >> 4) & 0xff
8
9 i += 1
10int main() {
11 int h, i, j, k;
12
13 for (h = 0; h < 256; h++) {
14 for (i = 0; i < 256; i++) {
15 for (j = 0; j < 256; j++) {
16 for (k = 0; k < 256; k++) {
17 printf("%d.%d.%d.%d\n", h, i, j, k);
18 }
19 }
20 }
21 }
22 return 0;
23}
24
I was wondering if there is a better way to do this? Both of these implementations allow 0.0.0.0 and private ips to be valid, so these are not solutions I want at the moment.
ANSWER
Answered 2021-Oct-26 at 06:50You can use ipaddress, it's in the standard library. is_private will exclude RFC1918 addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) as well as some others.
1i = 0
2while(i != -1):
3
4 b1 = (i >> 24) & 0xff
5 b2 = (i >> 16) & 0xff
6 b3 = (i >> 8) & 0xff
7 b4 = (i >> 4) & 0xff
8
9 i += 1
10int main() {
11 int h, i, j, k;
12
13 for (h = 0; h < 256; h++) {
14 for (i = 0; i < 256; i++) {
15 for (j = 0; j < 256; j++) {
16 for (k = 0; k < 256; k++) {
17 printf("%d.%d.%d.%d\n", h, i, j, k);
18 }
19 }
20 }
21 }
22 return 0;
23}
24import ipaddress
25
26all_ipv4 = ipaddress.ip_network('0.0.0.0/0')
27
28for host in all_ipv4.hosts():
29 if host.is_private:
30 continue
31 print(host) #do your thing here
32
QUESTION
Connect system which includes OPC UA IoT Agent, Orion Context Broker. Cygnus and Historic data with Postgres
Asked 2021-Oct-25 at 12:39I am trying to set up a system to save historic data with the flow like this: Prosys OPC-UA Server Simulation -> OPC-UA FIWARE IoT Agent -> Orion Context Broker -> FIWARE Cygnus Connector -> PostgreSQL database.
Here is the document I used to compose the docker-compose
file:
Here is the docker-compose
and .env
file I used
docker-compose.yml
1version: "3"
2#secrets:
3# age_idm_auth:
4# file: age_idm_auth.txt
5
6services:
7 iotage:
8 hostname: iotage
9 image: iotagent4fiware/iotagent-opcua:1.3.4
10 networks:
11 - hostnet
12 - iotnet
13 ports:
14 - "${AGENT_SERVER_PORT}:${AGENT_SERVER_PORT}"
15 - "4081:8080"
16 extra_hosts:
17 - "iotcarsrv:192.168.50.167"
18 - "HP:192.168.50.167"
19 depends_on:
20 - iotmongo
21 - orion
22 volumes:
23 - ./AGECONF:/opt/iotagent-opcua/conf
24 - ./certificates/charm:/opt/iotagent-opcua/certificates
25 command: /usr/bin/tail -f /var/log/lastlog
26
27 iotmongo:
28 hostname: iotmongo
29 image: mongo:3.4
30 networks:
31 - iotnet
32 volumes:
33 - iotmongo_data:/data/db
34 - iotmongo_conf:/data/configdb
35
36 ################ OCB ################
37
38 orion:
39 hostname: orion
40 image: fiware/orion:latest
41 networks:
42 - hostnet
43 - ocbnet
44 ports:
45 - "${ORION_PORT}:${ORION_PORT}"
46 depends_on:
47 - orion_mongo
48 #command: -dbhost mongo
49 entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
50
51 orion_mongo:
52 hostname: orion_mongo
53 image: mongo:3.4
54 networks:
55 ocbnet:
56 aliases:
57 - mongo
58 volumes:
59 - orion_mongo_data:/data/db
60 - orion_mongo_conf:/data/configdb
61 command: --nojournal
62
63 ############### CYGNUS ###############
64
65 cygnus:
66 image: fiware/cygnus-ngsi:${CYGNUS_VERSION}
67 hostname: cygnus
68 container_name: fiware-cygnus
69 networks:
70 - hostnet
71 depends_on:
72 - postgres-db
73 expose:
74 - "${CYGNUS_POSTGRESQL_SERVICE_PORT}" # 5055
75 - "${CYGNUS_API_PORT}" # 5080
76 ports:
77 - "${CYGNUS_POSTGRESQL_SERVICE_PORT}:${CYGNUS_POSTGRESQL_SERVICE_PORT}"
78 - "${CYGNUS_API_PORT}:${CYGNUS_API_PORT}"
79 environment:
80 - "CYGNUS_POSTGRESQL_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}"
81 - "CYGNUS_POSTGRESQL_HOST=postgres-db" # Hostname of the PostgreSQL server used to persist historical contex
82 - "CYGNUS_POSTGRESQL_PORT=${POSTGRES_DB_PORT}" # Port that the PostgreSQL server uses to listen to commands
83 - "CYGNUS_POSTGRESQL_DATABASE=postgres"
84 - "CYGNUS_POSTGRESQL_USER=postgres" # Username for the PostgreSQL database user
85 - "CYGNUS_POSTGRESQL_PASS=password" # Password for the PostgreSQL database user
86 - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true" # Switch to enable caching within the PostgreSQL configuration
87 - "CYGNUS_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}" # Notification Port that Cygnus listens when subcr
88 - "CYGNUS_API_PORT=${CYGNUS_API_PORT}" # Port that Cygnus listens on for operational reasons
89 - "CYGNUS_LOG_LEVEL=DEBUG" # The logging level for Cygnus
90
91
92 postgres-db:
93 image: postgres
94 hostname: postgres-db
95 expose:
96 - "${POSTGRES_DB_PORT}"
97 ports:
98 - "${POSTGRES_DB_PORT}:${POSTGRES_DB_PORT}"
99 networks:
100 - hostnet
101 environment:
102 - "POSTGRES_PASSWORD=password"
103 - "POSTGRES_USER=postgres"
104 - "POSTGRES_DB=postgres"
105 volumes:
106 - postgres-db:/var/lib/postgresql/data
107
108volumes:
109 iotmongo_data:
110 iotmongo_conf:
111 orion_mongo_data:
112 orion_mongo_conf:
113 postgres-db:
114
115networks:
116 hostnet:
117 iotnet:
118 ocbnet:
119
.env
1version: "3"
2#secrets:
3# age_idm_auth:
4# file: age_idm_auth.txt
5
6services:
7 iotage:
8 hostname: iotage
9 image: iotagent4fiware/iotagent-opcua:1.3.4
10 networks:
11 - hostnet
12 - iotnet
13 ports:
14 - "${AGENT_SERVER_PORT}:${AGENT_SERVER_PORT}"
15 - "4081:8080"
16 extra_hosts:
17 - "iotcarsrv:192.168.50.167"
18 - "HP:192.168.50.167"
19 depends_on:
20 - iotmongo
21 - orion
22 volumes:
23 - ./AGECONF:/opt/iotagent-opcua/conf
24 - ./certificates/charm:/opt/iotagent-opcua/certificates
25 command: /usr/bin/tail -f /var/log/lastlog
26
27 iotmongo:
28 hostname: iotmongo
29 image: mongo:3.4
30 networks:
31 - iotnet
32 volumes:
33 - iotmongo_data:/data/db
34 - iotmongo_conf:/data/configdb
35
36 ################ OCB ################
37
38 orion:
39 hostname: orion
40 image: fiware/orion:latest
41 networks:
42 - hostnet
43 - ocbnet
44 ports:
45 - "${ORION_PORT}:${ORION_PORT}"
46 depends_on:
47 - orion_mongo
48 #command: -dbhost mongo
49 entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
50
51 orion_mongo:
52 hostname: orion_mongo
53 image: mongo:3.4
54 networks:
55 ocbnet:
56 aliases:
57 - mongo
58 volumes:
59 - orion_mongo_data:/data/db
60 - orion_mongo_conf:/data/configdb
61 command: --nojournal
62
63 ############### CYGNUS ###############
64
65 cygnus:
66 image: fiware/cygnus-ngsi:${CYGNUS_VERSION}
67 hostname: cygnus
68 container_name: fiware-cygnus
69 networks:
70 - hostnet
71 depends_on:
72 - postgres-db
73 expose:
74 - "${CYGNUS_POSTGRESQL_SERVICE_PORT}" # 5055
75 - "${CYGNUS_API_PORT}" # 5080
76 ports:
77 - "${CYGNUS_POSTGRESQL_SERVICE_PORT}:${CYGNUS_POSTGRESQL_SERVICE_PORT}"
78 - "${CYGNUS_API_PORT}:${CYGNUS_API_PORT}"
79 environment:
80 - "CYGNUS_POSTGRESQL_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}"
81 - "CYGNUS_POSTGRESQL_HOST=postgres-db" # Hostname of the PostgreSQL server used to persist historical contex
82 - "CYGNUS_POSTGRESQL_PORT=${POSTGRES_DB_PORT}" # Port that the PostgreSQL server uses to listen to commands
83 - "CYGNUS_POSTGRESQL_DATABASE=postgres"
84 - "CYGNUS_POSTGRESQL_USER=postgres" # Username for the PostgreSQL database user
85 - "CYGNUS_POSTGRESQL_PASS=password" # Password for the PostgreSQL database user
86 - "CYGNUS_POSTGRESQL_ENABLE_CACHE=true" # Switch to enable caching within the PostgreSQL configuration
87 - "CYGNUS_SERVICE_PORT=${CYGNUS_POSTGRESQL_SERVICE_PORT}" # Notification Port that Cygnus listens when subcr
88 - "CYGNUS_API_PORT=${CYGNUS_API_PORT}" # Port that Cygnus listens on for operational reasons
89 - "CYGNUS_LOG_LEVEL=DEBUG" # The logging level for Cygnus
90
91
92 postgres-db:
93 image: postgres
94 hostname: postgres-db
95 expose:
96 - "${POSTGRES_DB_PORT}"
97 ports:
98 - "${POSTGRES_DB_PORT}:${POSTGRES_DB_PORT}"
99 networks:
100 - hostnet
101 environment:
102 - "POSTGRES_PASSWORD=password"
103 - "POSTGRES_USER=postgres"
104 - "POSTGRES_DB=postgres"
105 volumes:
106 - postgres-db:/var/lib/postgresql/data
107
108volumes:
109 iotmongo_data:
110 iotmongo_conf:
111 orion_mongo_data:
112 orion_mongo_conf:
113 postgres-db:
114
115networks:
116 hostnet:
117 iotnet:
118 ocbnet:
119# Orion
120ORION_PORT=1026
121
122# PostgreSQL
123POSTGRES_DB_PORT=5432
124
125# OPCUA IoT Agent
126AGENT_SERVER_PORT=4001
127
128# Cygnus
129CYGNUS_VERSION=2.10.0
130CYGNUS_API_PORT=5080
131CYGNUS_MYSQL_SERVICE_PORT=5050
132CYGNUS_MONGO_SERVICE_PORT=5051
133CYGNUS_CKAN_SERVICE_PORT=5052
134CYGNUS_HDFS_SERVICE_PORT=5053
135CYGNUS_CARTO_SERVICE_PORT=5054
136CYGNUS_POSTGRESQL_SERVICE_PORT=5055
137CYGNUS_ORION_SERVICE_PORT=5056
138CYGNUS_POSTGIS_SERVICE_PORT=5057
139CYGNUS_ELASTICSEARCH_SERVICE_PORT=5058
140CYGNUS_ARCGIS_SERVICE_PORT=5059
141
I can see the system is up and no error informed from docker
docker ps -a
result
However, when I use pgAdmin or docker exec
to examine the postgres database there was nothing in it.
\dt
command of psql gave me Did not find any relation
which means there was nothing logged into the database.
Please provide help to combine these two components of FIWARE (the IoT OPCUA agent and the Cygnus). It was also not clear for me if the Cygnus connector would create the database for us.
[EDIT]
The component seems to be working:
Simulation Server
Cynus only infor me that 'Configuration has no change', when I used docker exec
to get into the container, from Cygnus I was able to ping both orion, postgresql database containers
In postgresql: I don't see the desired database or table
My system from the Simulation Server -> OPCUA IoT Agent -> Orion was working well because from docker logs
I was still able to get the updated value of Simulation Server:
ANSWER
Answered 2021-Sep-01 at 10:31You have a complex scenario here, composed of an end-to-end chain of 5 components:
- Prosys OPC-UA Server Simulation
- OPC-UA FIWARE IoT Agent
- Orion Context Broker
- FIWARE Cygnus Connector
- PostgreSQL database
My recomendation here would be to check every step in the chain (looking in logs, etc.) to ensure everything is correct before of checking the next step.
QUESTION
DRF testing - create object with a many-to-many relation
Asked 2021-Oct-20 at 15:20I have the following model Project:
1class Project(models.Model):
2 name = models.CharField(max_length=128)
3 slug = models.SlugField(blank=True)
4 assigned_to = models.ManyToManyField(
5 User, blank=True, related_name="assignees")
6 created_date = models.DateField(auto_now_add=True)
7 updated_date = models.DateField(auto_now_add=False, auto_now=True)
8
If I need to create a new project, all I need to do is supply the name
alone. This works for both the Admin dashboard and DRF APIVIEW. But when I try to test the functionality with DRF with an API call, I get the error: [b'{"assigned_to":["This field is required."]}']
Although the field is not required.
My test code below
1class Project(models.Model):
2 name = models.CharField(max_length=128)
3 slug = models.SlugField(blank=True)
4 assigned_to = models.ManyToManyField(
5 User, blank=True, related_name="assignees")
6 created_date = models.DateField(auto_now_add=True)
7 updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21 client = APIClient()
22 project = None
23
24 name = 'IOT on Blockchain'
25 dead_line = datetime.date(2021, 11, 21)
26 data = {
27 'name': name,
28 'dead_line': dead_line,
29 }
30
31 def create_user(self):
32 username = 'test_user1'
33 email = 'test.user1@gmail.com'
34 password = '@1234xyz@'
35 user_type = 'regular'
36 data = {'username': username,
37 'email': email,
38 'password': password,
39 'user_type': user_type,
40 }
41 return User.objects.create_user(**data)
42
43 def create_project(self):
44 project = Project.objects.create(**self.data)
45 user = self.create_user()
46 project.assigned_to.add(user)
47 return project
48
49 def test_create_project_without_api(self):
50 """
51 Ensure we can create a new project object.
52 """
53 self.project = self.create_project()
54 self.assertEqual(Project.objects.count(), 1)
55 self.assertEqual(self.project.name, 'IOT on Blockchain')
56 self.assertEqual(self.project.dead_line,
57 datetime.date(2021, 11, 21))
58 self.assertFalse(self.project.reached_deadline)
59 self.assertEqual(self.project.days_to_deadline, 7)
60 # runs successfully
61
62 def test_create_project_with_api(self):
63 """
64 Ensure we can create a new project object with an
65 API call.
66 """
67 url = 'http://127.0.0.1:8000/api/projects'
68 project = self.client.post(url, self.data, format='json')
69 # project.data.assigned_to.set(self.create_user())
70 pprint(project.__dict__)
71 self.assertEqual(Project.objects.count(), 1)
72 self.assertEqual(self.project.name, 'IOT on Blockchain')
73 self.assertEqual(self.project.slug, 'iot-on-blockchain')
74 # does not run successfully (error mentioned in text body)
75
76 def test_delete_project(self):
77 """
78 We can delete a user
79 """
80 self.project = self.create_project()
81 self.project.delete()
82 self.assertEqual(Project.objects.count(), 0)
83
Edit: Added serializer code
1class Project(models.Model):
2 name = models.CharField(max_length=128)
3 slug = models.SlugField(blank=True)
4 assigned_to = models.ManyToManyField(
5 User, blank=True, related_name="assignees")
6 created_date = models.DateField(auto_now_add=True)
7 updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21 client = APIClient()
22 project = None
23
24 name = 'IOT on Blockchain'
25 dead_line = datetime.date(2021, 11, 21)
26 data = {
27 'name': name,
28 'dead_line': dead_line,
29 }
30
31 def create_user(self):
32 username = 'test_user1'
33 email = 'test.user1@gmail.com'
34 password = '@1234xyz@'
35 user_type = 'regular'
36 data = {'username': username,
37 'email': email,
38 'password': password,
39 'user_type': user_type,
40 }
41 return User.objects.create_user(**data)
42
43 def create_project(self):
44 project = Project.objects.create(**self.data)
45 user = self.create_user()
46 project.assigned_to.add(user)
47 return project
48
49 def test_create_project_without_api(self):
50 """
51 Ensure we can create a new project object.
52 """
53 self.project = self.create_project()
54 self.assertEqual(Project.objects.count(), 1)
55 self.assertEqual(self.project.name, 'IOT on Blockchain')
56 self.assertEqual(self.project.dead_line,
57 datetime.date(2021, 11, 21))
58 self.assertFalse(self.project.reached_deadline)
59 self.assertEqual(self.project.days_to_deadline, 7)
60 # runs successfully
61
62 def test_create_project_with_api(self):
63 """
64 Ensure we can create a new project object with an
65 API call.
66 """
67 url = 'http://127.0.0.1:8000/api/projects'
68 project = self.client.post(url, self.data, format='json')
69 # project.data.assigned_to.set(self.create_user())
70 pprint(project.__dict__)
71 self.assertEqual(Project.objects.count(), 1)
72 self.assertEqual(self.project.name, 'IOT on Blockchain')
73 self.assertEqual(self.project.slug, 'iot-on-blockchain')
74 # does not run successfully (error mentioned in text body)
75
76 def test_delete_project(self):
77 """
78 We can delete a user
79 """
80 self.project = self.create_project()
81 self.project.delete()
82 self.assertEqual(Project.objects.count(), 0)
83class ProjectWriteSerializer(serializers.ModelSerializer):
84 """
85 This serializer is used for CREATE, UPDATE operations on the Project model.
86 """
87
88 # We receive list of user ids (ids[int] <= 0) by which we assign
89 # users to a project
90 assigned_to = serializers.PrimaryKeyRelatedField(
91 queryset=User.objects.all(), many=True)
92
93 class Meta:
94 model = Project
95 fields = ('id', 'name', 'slug', 'assigned_to')
96
Any insights and help is very appreciated.
ANSWER
Answered 2021-Oct-20 at 15:20You have specified your own serializer field. As a result, it will no longer look at the blank=True
part, and by default serializer fields are required. You can make these optional with:
1class Project(models.Model):
2 name = models.CharField(max_length=128)
3 slug = models.SlugField(blank=True)
4 assigned_to = models.ManyToManyField(
5 User, blank=True, related_name="assignees")
6 created_date = models.DateField(auto_now_add=True)
7 updated_date = models.DateField(auto_now_add=False, auto_now=True)
8import datetime
9from marshmallow import pprint
10
11from rest_framework.test import APITestCase, APIClient
12from freezegun import freeze_time
13
14from accounts.models import User
15from .models import Project
16
17
18@freeze_time("2021-11-14")
19class ProjectTests(APITestCase):
20
21 client = APIClient()
22 project = None
23
24 name = 'IOT on Blockchain'
25 dead_line = datetime.date(2021, 11, 21)
26 data = {
27 'name': name,
28 'dead_line': dead_line,
29 }
30
31 def create_user(self):
32 username = 'test_user1'
33 email = 'test.user1@gmail.com'
34 password = '@1234xyz@'
35 user_type = 'regular'
36 data = {'username': username,
37 'email': email,
38 'password': password,
39 'user_type': user_type,
40 }
41 return User.objects.create_user(**data)
42
43 def create_project(self):
44 project = Project.objects.create(**self.data)
45 user = self.create_user()
46 project.assigned_to.add(user)
47 return project
48
49 def test_create_project_without_api(self):
50 """
51 Ensure we can create a new project object.
52 """
53 self.project = self.create_project()
54 self.assertEqual(Project.objects.count(), 1)
55 self.assertEqual(self.project.name, 'IOT on Blockchain')
56 self.assertEqual(self.project.dead_line,
57 datetime.date(2021, 11, 21))
58 self.assertFalse(self.project.reached_deadline)
59 self.assertEqual(self.project.days_to_deadline, 7)
60 # runs successfully
61
62 def test_create_project_with_api(self):
63 """
64 Ensure we can create a new project object with an
65 API call.
66 """
67 url = 'http://127.0.0.1:8000/api/projects'
68 project = self.client.post(url, self.data, format='json')
69 # project.data.assigned_to.set(self.create_user())
70 pprint(project.__dict__)
71 self.assertEqual(Project.objects.count(), 1)
72 self.assertEqual(self.project.name, 'IOT on Blockchain')
73 self.assertEqual(self.project.slug, 'iot-on-blockchain')
74 # does not run successfully (error mentioned in text body)
75
76 def test_delete_project(self):
77 """
78 We can delete a user
79 """
80 self.project = self.create_project()
81 self.project.delete()
82 self.assertEqual(Project.objects.count(), 0)
83class ProjectWriteSerializer(serializers.ModelSerializer):
84 """
85 This serializer is used for CREATE, UPDATE operations on the Project model.
86 """
87
88 # We receive list of user ids (ids[int] <= 0) by which we assign
89 # users to a project
90 assigned_to = serializers.PrimaryKeyRelatedField(
91 queryset=User.objects.all(), many=True)
92
93 class Meta:
94 model = Project
95 fields = ('id', 'name', 'slug', 'assigned_to')
96class ProjectWriteSerializer(serializers.ModelSerializer):
97 assigned_to = serializers.PrimaryKeyRelatedField(
98 queryset=User.objects.all(),
99 many=True,
100 <strong>required=True</strong>
101 )
102
103 class Meta:
104 model = Project
105 fields = ('id', 'name', 'slug', 'assigned_to')
QUESTION
Postgres March stange results
Asked 2021-Oct-20 at 13:37I'm tring to calculate time prorata between two dates over multi month periods using SQL on PostgresSQL.
My formula is pretty simple
1select iot.*,
2 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF( extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4 select DATE '2020-05-01' as DATE_DEBUT, DATE '2020-05-31' as DATE_FIN
5) iot
6
Give me expected result => 1
But when I switch to march with
1select iot.*,
2 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF( extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4 select DATE '2020-05-01' as DATE_DEBUT, DATE '2020-05-31' as DATE_FIN
5) iot
6select iot.*,
7 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
8from (
9 select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
10) iot
11
I've got a weird result of 1.0344827586206897
Any ideas?
ANSWER
Answered 2021-Oct-20 at 13:37This is caused by date_trunc('month'
, this function returns a date with a timezone, but your whole calculus doesn't handle the timezone.
If you force Postgres to ignore the timezone, it should work:
1select iot.*,
2 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF( extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
3from (
4 select DATE '2020-05-01' as DATE_DEBUT, DATE '2020-05-31' as DATE_FIN
5) iot
6select iot.*,
7 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)), 0::double precision) as prorata
8from (
9 select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
10) iot
11select iot.*,
12 (iot.DATE_FIN - iot.DATE_DEBUT)::double precision / NULLIF(extract (day from iot.DATE_FIN - date_trunc('month', iot.DATE_DEBUT)::timestamp without time zone), 0::double precision) as prorata
13from (
14 select DATE '2020-03-01' as DATE_DEBUT, DATE '2020-03-31' as DATE_FIN
15) iot
16
I just cast date_trunc
with ::timestamp without time zone
.
Community Discussions contain sources that include Stack Exchange Network
Tutorials and Learning Resources in IOT
Tutorials and Learning Resources are not available at this moment for IOT