hydra | OAuth Provider written in Go - cloud | Authentication library
kandi X-RAY | hydra Summary
Support
Quality
Security
License
Reuse
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample Here
hydra Key Features
hydra Examples and Code Snippets
Trending Discussions on hydra
Trending Discussions on hydra
QUESTION
I have downloaded the corpus of articles Aminar DBLP Version 11. The corpus is a huge text file (12GB) which each line is a self-contained JSON string:
'{"id": "100001334", "title": "Ontologies in HYDRA - Middleware for Ambient Intelligent Devices.", "authors": [{"name": "Peter Kostelnik", "id": "2702511795"}, {"name": "Martin Sarnovsky", "id": "2041014688"}, {"name": "Jan Hreno", "id": "2398560122"}], "venue": {"raw": "AMIF"}, "year": 2009, "n_citation": 2, "page_start": "43", "page_end": "46", "doc_type": "", "publisher": "", "volume": "", "issue": "", "fos": [{"name": "Lernaean Hydra", "w": 0.4178039}, {"name": "Database", "w": 0.4269269}, {"name": "World Wide Web", "w": 0.415332377}, {"name": "Ontology (information science)", "w": 0.459045082}, {"name": "Computer science", "w": 0.399807781}, {"name": "Middleware", "w": 0.5905041}, {"name": "Ambient intelligence", "w": 0.5440575}]}'
All JSON strings are new line separated.
When I open the file using PySpark, it returns a dataframe with one column containing JSON strings:
df = spark.read.text(path_to_data)
df.show()
+--------------------+
| value|
+--------------------+
|{"id": "100001334...|
|{"id": "100001888...|
|{"id": "100002270...|
|{"id": "100004108...|
|{"id": "10000571"...|
|{"id": "100007563...|
|{"id": "100008278...|
|{"id": "100008490...|
I need to access JSON fields to build my deep learning model.
My first attempt was trying to open the file using JSON method as mentioned in this question:
df = spark.read.option("wholeFile", True).option("mode", "PERMISSIVE").json(path_to_data)
But all the proposed solutions took ages to run (more than 3h) with no results to show.
My second attempt was trying to parse a JSON object from JSON string to get a dataframe with columns as follows:
df = spark.read.text(path_to_data)
schema = StructType([StructField("id", StringType()), StructField("title", StringType()), StructField("authors", ArrayType(MapType(StringType(), StringType()))), StructField("venue", MapType(StringType(), StringType()), True), StructField("year", IntegerType(), True), StructField("keywords", ArrayType(StringType()), True), StructField("references", ArrayType(StringType()), True), StructField("n_citation", IntegerType(), True), StructField("page_start", StringType(), True), StructField("page_end", StringType(), True), StructField("doc_type", StringType(), True), StructField("lang", StringType(), True), StructField("publisher", StringType(), True), StructField("volume", StringType(), True), StructField("issue", StringType(), True), StructField("issn", StringType(), True), StructField("isbn", StringType(), True), StructField("doi", StringType(), True), StructField("pdf", StringType(), True), StructField("url", ArrayType(StringType()), True),
StructField("abstract", StringType(), True), StructField("indexed_abstract", StringType(), True)])
datajson = df.withColumn("jsonData", from_json(col("value"),schema)).select("jsonData.*")
But it returned the exception "cannot resolve column due to data type mismatch PySpark", even though the data types of each field in the schema are true (based on the official website of corpus here)
My third attempt was trying to parse the JSON string to Map data type:
casted = df.withColumn("value", from_json(df.value, MapType(StringType(),StringType())))
It gave me the following result:
root
|-- value: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
+--------------------+
| value|
+--------------------+
|{id -> 100001334,...|
|{id -> 1000018889...|
|{id -> 1000022707...|
|{id -> 100004108,...|
|{id -> 10000571, ...|
|{id -> 100007563,...|
|{id -> 100008278,...|
Now, each row is a valid JSON object which can be accessed as follows:
row = casted.first()
row.value['id']
row.value['title']
row.value['authors']
Now, my question is how to convert this dataframe of one column named 'value' to a dataframe with the columns mentioned above (id, title, authors, etc) based on JSON objects?
ANSWER
Answered 2022-Mar-23 at 13:51Reading the file without providing the schema is taking longer time. I tried to split the huge file in smaller chunks to understand the schema and it failed with Found duplicate column(s) in the data schema:
I tried the below approach on the same dataset with provided schema and it worked.
from pyspark.sql.types import StructType,StructField,StringType,IntegerType,ArrayType
schema = StructType([StructField("id", StringType()), StructField("title", StringType()), StructField("authors", ArrayType(MapType(StringType(), StringType()))), StructField("venue", MapType(StringType(), StringType()), True), StructField("year", IntegerType(), True), StructField("keywords", ArrayType(StringType()), True), StructField("references", ArrayType(StringType()), True), StructField("n_citation", IntegerType(), True), StructField("page_start", StringType(), True), StructField("page_end", StringType(), True), StructField("doc_type", StringType(), True), StructField("lang", StringType(), True), StructField("publisher", StringType(), True), StructField("volume", StringType(), True), StructField("issue", StringType(), True), StructField("issn", StringType(), True), StructField("isbn", StringType(), True), StructField("doi", StringType(), True), StructField("pdf", StringType(), True), StructField("url", ArrayType(StringType()), True),
StructField("abstract", StringType(), True), StructField("indexed_abstract", StringType(), True)])
df = spark.read.option("wholeFile", True).option("mode", "PERMISSIVE").schema(schema).json("dblp_papers_v11.txt")
df.show()
output
[![+----------+--------------------+--------------------+--------------------+----+--------+--------------------+----------+----------+--------+----------+----+--------------------+------+-----+----+----+--------------------+----+----+--------+--------------------+
| id| title| authors| venue|year|keywords| references|n_citation|page_start|page_end| doc_type|lang| publisher|volume|issue|issn|isbn| doi| pdf| url|abstract| indexed_abstract|
+----------+--------------------+--------------------+--------------------+----+--------+--------------------+----------+----------+--------+----------+----+--------------------+------+-----+----+----+--------------------+----+----+--------+--------------------+
| 100001334|Ontologies in HYD...|\[{name -> Peter K...| {raw -> AMIF}|2009| null| null| 2| 43| 46| |null| | | |null|null| null|null|null| null| null|
|1000018889|Remote Policy Enf...|\[{name -> Fabio M...|{raw -> internati...|2013| null|\[94181602, 150466...| 2| 70| 84|Conference|null| Springer, Cham| | |null|null|10.1007/978-3-319...|null|null| null|{"IndexLength":17...|
|1000022707|A SIMPLE OBSERVAT...|\[{name -> Jerzy M...|{raw -> Reports o...|2009| null|\[1972178849, 2069...| 0| 19| 29| Journal|null| | 44| |null|null| null|null|null| null|{"IndexLength":49...|
| 100004108|Gait based human ...|\[{name -> Emdad H...|{raw -> internati...|2012| null|\[1578000111, 2120...| 0| 319| 328|Conference|null|Springer, Berlin,...| | |null|null|10.1007/978-3-642...|null|null| null|{"IndexLength":82...|
| 10000571|The GAME Algorith...|\[{name -> Pavel K...|{raw -> internati...|2008| null|\[291899685, 19641...| 5| 859| 868|Conference|null|Springer, Berlin,...| | |null|null|10.1007/978-3-540...|null|null| null|{"IndexLength":17...|
| 100007563|Formal Verificati...|\[{name -> George ...|{raw -> Software ...|2006| null|\[1578963809, 1612...| 1| 650| 656| Journal|null| | | |null|null| null|null|null| null|{"IndexLength":87...|
| 100008278|EMOTIONAL AND RAT...|\[{name -> Colin G...|{raw -> internati...|2010| null|\[116282327, 14967...| 2| 238| |Conference|null| | | |null|null| null|null|null| null|{"IndexLength":12...|
| 100008490|Principle-Based P...|\[{name -> Sandiwa...|{raw -> Natural L...|1991| null| null| 3| 43| 60| |null| | | |null|null| null|null|null| null| null|][1]][1]
QUESTION
The documentation in https://hydra.cc/docs/patterns/select_multiple_configs_from_config_group/ shows how to pick multiple configs from a Config Group and place them in a dictionary-like structure.
However, as mentioned in the very last paragraph there, "example uses an explicit nesting level inside each of the configs to prevent them stepping over one another". For my use-case, this would prove extremely cumbersome and I would like to avoid it at all costs if possible.
Is there a way to achieve a similar result without resorting to explicitly adding the level in the individual configs? Thanks in advance :)
ANSWER
Answered 2022-Mar-06 at 14:58You can use a defaults-list @package
keyword to achieve a similar result.
Let's assume you have changed the yaml files server/site/fb.yaml
and server/site/google.yaml
to not contain the "explicit nesting", so e.g. server/site/fb.yaml
contains only the data domain: facebook.com
.
You can achieve the same output as from the docs webpage using the following defaults list in server/apache.yaml`:
# Option 1: With a non-overridable defaults list
defaults:
- site/fb@site.fb
- site/google@site.google
# Option 2: With an overridable defaults list
defaults:
- site@site.fb: fb
- site@site.google: google
Either option 1 or option 2 above produces this output:
$ python my_app.py
server:
site:
fb:
domain: facebook.com
google:
domain: google.com
host: localhost
port: 443
The @package
directive here could be any compound key that you want. For example, using the following defaults list:
# Option 1: With a non-overridable defaults list
defaults:
- site/fb@foo
- site/google@bar
# Option 2: With an overridable defaults list
defaults:
- site@foo: fb
- site@bar: google
We get this result:
$ python my_app.py
server:
foo:
domain: facebook.com
bar:
domain: google.com
host: localhost
port: 443
Using option 2 (an overridable defaults list) means you can override the given default option using the CLI:
$ python my_app.py server/site@server.foo=amazon
server:
foo:
domain: amazon.com
bar:
domain: google.com
host: localhost
port: 443
QUESTION
After upgrading to Api platform 2.6, the non ApiResource
entities started being serialized (ld-json) along with a fake identifier and their type. For example:
{
"@type": "MetaData",
"@id": "_:11294",
"id": "bf1e417c-27ff-48c1-a591-40e5a43c708c",
"key": "key1",
"value": "value1"
}
we would like to remove these fields. To do so I've tried to implement a custom normalizer hoping to be able to hook in after the object is transformed in array and before the array is converted to a json.
Looking at the registered normalizers:
----------------------------------------------------------- ---------- ---------------------------------------------------------------------------
Service ID priority Class name
----------------------------------------------------------- ---------- ---------------------------------------------------------------------------
api_platform.serializer.normalizer.item -895 ApiPlatform\Core\Serializer\ItemNormalizer
api_platform.problem.normalizer.constraint_violation_list -780 ApiPlatform\Core\Problem\Serializer\ConstraintViolationListNormalizer
api_platform.hydra.normalizer.collection_filters -985 ApiPlatform\Core\Hydra\Serializer\CollectionFiltersNormalizer
api_platform.hydra.normalizer.error -800 ApiPlatform\Core\Hydra\Serializer\ErrorNormalizer
api_platform.hydra.normalizer.entrypoint -800 ApiPlatform\Core\Hydra\Serializer\EntrypointNormalizer
api_platform.hydra.normalizer.constraint_violation_list -780 ApiPlatform\Core\Hydra\Serializer\ConstraintViolationListNormalizer
api_platform.hydra.normalizer.documentation -800 ApiPlatform\Core\Hydra\Serializer\DocumentationNormalizer
api_platform.jsonld.normalizer.object -995 ApiPlatform\Core\JsonLd\Serializer\ObjectNormalizer
api_platform.swagger.normalizer.api_gateway -780 ApiPlatform\Core\Swagger\Serializer\ApiGatewayNormalizer
api_platform.openapi.normalizer.api_gateway -780 ApiPlatform\Core\Swagger\Serializer\ApiGatewayNormalizer
api_platform.serializer.uuid_denormalizer ApiPlatform\Core\Bridge\RamseyUuid\Serializer\UuidDenormalizer
serializer.denormalizer.array -990 Symfony\Component\Serializer\Normalizer\ArrayDenormalizer
serializer.normalizer.object -1000 Symfony\Component\Serializer\Normalizer\ObjectNormalizer
serializer.normalizer.problem -890 Symfony\Component\Serializer\Normalizer\ProblemNormalizer
serializer.normalizer.json_serializable -900 Symfony\Component\Serializer\Normalizer\JsonSerializableNormalizer
serializer.normalizer.datetime -910 Symfony\Component\Serializer\Normalizer\DateTimeNormalizer
serializer.normalizer.data_uri -920 Symfony\Component\Serializer\Normalizer\DataUriNormalizer
serializer.normalizer.dateinterval -915 Symfony\Component\Serializer\Normalizer\DateIntervalNormalizer
serializer.normalizer.datetimezone -915 Symfony\Component\Serializer\Normalizer\DateTimeZoneNormalizer
serializer.normalizer.constraint_violation_list -915 Symfony\Component\Serializer\Normalizer\ConstraintViolationListNormalizer
api_platform.problem.normalizer.error -810 ApiPlatform\Core\Problem\Serializer\ErrorNormalizer
----------------------------------------------------------- ---------- ---------------------------------------------------------------------------
the closest I've got was around the api_platform.jsonld.normalizer.object
however if the priority I use is -994, I get the original object, if I use -996 I get the single string/boolean/numeric fields (so after the normalization).
Is there any way I can get the associative array so that I can remove @id and @type if @id starts with _:
?
The line of the code that adds those properties seems to be this however that probably just adds to the context it doesn't include those fields in the normalized array.
ANSWER
Answered 2022-Feb-20 at 09:54No idea how to decorate the serializer, but I will handle your issue with an event subscriber.
The "POST_SERIALIZE" priority contains the serialized response body, and you can fetch it and modify it.
Like this:
["removeFakeIdentifiersFromJsonLdResponseBody", EventPriorities::POST_SERIALIZE],
];
}
public function removeFakeIdentifiersFromJsonLdResponseBody(ViewEvent $event): void
{
$requestAcceptHeader = $event->getRequest()->headers->get('accept', 'none');
if (str_contains($requestAcceptHeader, "application/ld+json")) {
$this->removefakeIdentifiersFromResponseBody($event);
}
}
private function removefakeIdentifiersFromResponseBody(ViewEvent $event): void
{
$responseBody = $event->getControllerResult();
$decodedResponseBody = json_decode($responseBody);
$this->removeFakeIdentifiersFromObject($decodedResponseBody);
$encodedResponseBody = json_encode($decodedResponseBody);
$event->setControllerResult($encodedResponseBody);
}
private function removeFakeIdentifiersFromObject(object $responseBody): void
{
foreach ($responseBody as $property => $value) {
$this->removeFakeIdRecursively($property, $value, $responseBody);
}
}
private function removeFakeIdRecursively(string $property, mixed $value, object $responseBody): void
{
if ($property === "@id" && str_starts_with($value, "_:")) {
unset($responseBody->$property); // removes "@id"
unset($responseBody->{"@type"});
} elseif (is_object($value)) {
$this->removeFakeIdentifiersFromObject($value);
} elseif (is_array($value)) {
foreach ($value as $object) {
if (is_object($object)) {
$this->removeFakeIdentifiersFromObject($object);
}
}
}
}
}
QUESTION
Hydra provides a way to dynamically create a hierarchical configuration by composition and override it through config files and the command line, leveraging OmegaConf. I look for a recommended way to document the parameters but I could not find (a documented) one. What are best practices for that? Coming from argparse, I like the way of documenting the parameter inline, i.e. close to the code.
ANSWER
Answered 2022-Feb-15 at 02:47Excellent question! The answer is: parameter-by-parameter documentation is not yet implemented (as of Hydra v1.1, OmegaConf v2.1).
The future plans are:
- in OmegaConf, expose an API allowing users to attach documentation (and other metadata) to each field of a structured config. See this open OmegaConf issue.
- Once the above is complete, implement a Hydra feature allowing a parameter-specific help messages to be printed based on the parameter's metadata. See this open Hydra issue.
For now, the best we can do is to customize the general application help message (i.e. the --help
command line flag).
QUESTION
Please, see this question for context and the answer by MrC aka Shaun Curtis
This question is about something I've tried to solve in the past without great success. Are you familiar with this sample I've once downloaded and run it. It did not work. I then realized that I must add the base url to the url in the browser's address bar in order to run the first project, for instance: https://localhost: 44302/FirstApp
That is, the Client project. And for the SecondClient it should be https://localhost: 44302/SecondApp
. This is exactly how the sample app by MrC aka Shaun Curtis works, though he added a Razor Pages app to provide a menu for redirection to the four projects.
What I tried to do without much success is to make the first WebAssemby front end project, which is hosted, to be the default; that is when I run the app, or type in the address bar https://localhost: 44302.
And if I type https://localhost: 44302/FirstApp
I see the first stand alone WebAssembly project I added to the solution. And a second project, and a third project, and so on, all are WebAssembly projects. I could not do that: When I ran the default project everything is fine... I can navigate within the bounds of the project, route to the Counter page, FetchData page, etc.
But when I add the segment /FirstApp to the url in the address bar and hit enter, the Router displays the message "Sorry, there's nothing at this address." instead of navigating to the project represented by the base url /FirstApp/
Does anyone here have any idea how to achieve the requested feature I am looking for ?
ANSWER
Answered 2022-Feb-14 at 17:57This is a summary of the Repo that demonstrates how to do this.
The sub-folder Web Assembly projects have project files like this. The import bit is setting the
net6.0
enable
enable
grey
....
and Index.html
like this. We've updated the paths on the base
, css and the framework js file.
The Web Project has dependencies on all the Web Assembly projects, so they can all be mapped to wwwwroot
.
The Web Project Program
looks like this with specific end points for each Web Assembly SPA. The default maps to the base Web Assembly project - Blazr.Medusa.WASM.
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddRazorPages();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
app.UseExceptionHandler("/Error");
// The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
app.UseHttpsRedirection();
app.UseStaticFiles();
app.MapWhen(ctx => ctx.Request.Path.StartsWithSegments("/grey"), app1 =>
{
app1.UseBlazorFrameworkFiles("/grey");
app1.UseRouting();
app1.UseEndpoints(endpoints =>
{
endpoints.MapFallbackToFile("/grey/{*path:nonfile}", "/grey/index.html");
});
});
app.MapWhen(ctx => ctx.Request.Path.StartsWithSegments("/green"), app1 =>
{
app1.UseBlazorFrameworkFiles("/green");
app1.UseRouting();
app1.UseEndpoints(endpoints =>
{
endpoints.MapFallbackToFile("/green/{*path:nonfile}", "/green/index.html");
});
});
app.MapWhen(ctx => ctx.Request.Path.StartsWithSegments("/purple"), app1 =>
{
app1.UseBlazorFrameworkFiles("/purple");
app1.UseRouting();
app1.UseEndpoints(endpoints =>
{
endpoints.MapFallbackToFile("/purple/{*path:nonfile}", "/purple/index.html");
});
});
app.UseBlazorFrameworkFiles("");
app.UseRouting();
app.UseAuthorization();
app.MapRazorPages();
app.MapFallbackToFile("/index.html");
app.Run();
The Site Links component in the MainLayout
of each site provides navigation between the SPA's
Go Base
Go Grey
Go Green
Go Purple
@code {
[Inject] private NavigationManager? NavManager { get; set; }
private void Go(string colour)
=> this.NavManager?.NavigateTo($"/{colour}", true);
}
QUESTION
Recently I have started to use hydra to manage the configs in my application. I use Structured Configs to create schema for .yaml config files. Structured Configs in Hyda uses dataclasses for type checking. However, I also want to use some kind of validators for some of the parameter I specify in my Structured Configs (something like this).
Do you know if it is somehow possible to use Pydantic for this purpose? When I try to use Pydantic, OmegaConf complains about it:
omegaconf.errors.ValidationError: Input class 'SomeClass' is not a structured config. did you forget to decorate it as a dataclass?
ANSWER
Answered 2022-Jan-10 at 05:58See pydantic.dataclasses.dataclass, which are a drop-in replacement for the standard-library dataclasses with some extra type-checking.
QUESTION
In Hydra I have the following configuration:
βββ conf
β βββ config.yaml
β βββ callbacks
β β βββ callback_01.yaml
β β βββ callback_02.yaml
β βββ trainer
β βββ default.yaml
The callbacks have a structure like this:
_target_: callback_to_instantiate
I need to pass to the trainer/default.yaml both the callbacks through interpolation. I tried like this:
_target_: pytorch_lightning.Trainer
callbacks:
- ${callbacks.callback_01}
- ${callbacks.callback_02}
With the config.yaml like this:
defaults:
- _self_
- trainer: default
I did also other trials but it doesn't seem to work. Is there a way to interpolate like that in a yaml file by using two or more yaml files that are in the config group? I would like if possible to keep this structure.
ANSWER
Answered 2022-Feb-10 at 19:48Currently the recommended approach is:
- compose a mapping whose values are the desired callbacks, and then
- use the
oc.dict.values
OmegaConf resolver to get a list of values from that dictionary.
# conf/config.yaml
defaults:
- callbacks@_callback_dict.cb1: callback_01
- callbacks@_callback_dict.cb2: callback_02
- trainer: default
- _self_
# conf/trainer/default.yaml
_target_: pytorch_lightning.Trainer
callbacks: ${oc.dict.values:_callback_dict}
# my_app.py
from typing import Any
import hydra
from omegaconf import DictConfig, OmegaConf
@hydra.main(config_path="conf", config_name="config")
def app(cfg: DictConfig) -> Any:
OmegaConf.resolve(cfg)
del cfg._callback_dict
print(OmegaConf.to_yaml(cfg))
if __name__ == "__main__":
app()
At the command line:
$ python my_app.py
trainer:
_target_: pytorch_lightning.Trainer
callbacks:
- _target_: callback_to_instantiate_01
- _target_: callback_to_instantiate_02
For reference, there is an open issue on Hydra's github repo advocating for an improved user experience around
QUESTION
I have a project where I want to build a full-blown IDP (using Golang). So technically, a user wants to SSO into another system using my service. I am looking to build this service from scratch. Upon researching for open-source IDP solutions, I came across ory/Hydra and ory/Kratos. I went through their documentation and did a quick-start tutorial. I am still confused about which of the above 2 libraries are suitable for the development of this service.
From a high-level standpoint, this is what I am trying to do.
- There is an existing third-party web app X.
- A user has already login credentials for my service Y.
- A user wants to post some comments in app X.
- He/She is redirected to my service Y login screen if not logged in.
- After login, the Callback URL returns him back to app X comment section.
ANSWER
Answered 2022-Feb-09 at 19:16ory/Kratos is not what you are looking for because it is designed to answer your user management basic needs.
The right tool for you is ory/Hydra. I have copied this from its documentation:
If you want apps and websites you don't own to use your application as a potential sign in (e.g. be listed alongside "Sign in with Google", "Sign in with Apple"), ORY Hydra is the right tool for you.
source: https://www.ory.sh/hydra/docs/concepts/before-oauth2
QUESTION
Is there a better way to reload a hydra config from an experiment with enumerations? Right now I reload it like so:
initialize_config_dir(config_dir=exp_dir, ".hydra"), job_name=config_name)
cfg = compose(config_name, overrides=overrides)
print(cfg.enum)
>>> ENUM1
But ENUM1 is actually an enumeration that normally loads as
>>>
I am able to fix this by adding a configstore default to the experiment hydra file:
defaults:
- base_config_cs
Which now results in
initialize_config_dir(config_dir=exp_dir, ".hydra"), job_name=config_name)
cfg = compose(config_name, overrides=overrides)
print(cfg.enum)
>>>
Is there a better way to do this without adding this? Or can I add the default in the python code?
ANSWER
Answered 2022-Feb-05 at 13:52This is a good question -- reliably reloading configs from previous Hydra runs is an area that could be improved. As you've discovered, loading the saved file config.yaml
directly results in an untyped DictConfig object.
The solution below involves a script called reload.py
that creates a config node with a defaults list that loads both the schema base_config_cs
and the saved file config.yaml
.
At the end of this post I also give a simple solution that involves loading .hydra/overrides.yaml
to re-run the config composition process.
Suppose you've run a Hydra job with the following setup:
# app.py
from dataclasses import dataclass
from enum import Enum
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import DictConfig
class SomeEnumClass(Enum):
ENUM1 = 1
ENUM2 = 2
@dataclass
class Schema:
enum: SomeEnumClass
x: int = 123
y: str = "abc"
def store_schema() -> None:
cs = ConfigStore.instance()
cs.store(name="base_config_cs", node=Schema)
@hydra.main(config_path=".", config_name="foo")
def app(cfg: DictConfig) -> None:
print(cfg)
if __name__ == "__main__":
store_schema()
app()
# foo.yaml
defaults:
- base_config_cs
- _self_
enum: ENUM1
x: 456
$ python app.py y=xyz
{'enum': , 'x': 456, 'y': 'xyz'}
After running app.py
, there exists a directory outputs/2022-02-05/06-42-42/.hydra
containing the saved file config.yaml
.
As you correctly pointed out in your question, to reload the saved config you must merge the schema base_config_cs
with the contents of config.yaml
. Here is a pattern for accomplishing that:
# reload.py
import os
from hydra import compose, initialize_config_dir
from hydra.core.config_store import ConfigStore
from app import store_schema
config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-19-56")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
assert os.path.exists(f"{saved_cfg_dir}/{config_name}.yaml")
store_schema() # stores `base_config_cs`
cs = ConfigStore.instance()
cs.store(
name="reload_conf",
node={
"defaults": [
"base_config_cs",
config_name,
]
},
)
with initialize_config_dir(config_dir=saved_cfg_dir):
cfg = compose("reload_conf")
print(cfg)
$ python reload.py
{'enum': , 'x': 456, 'y': 'xyz'}
In the above, python file reload.py
, we store a node called reload_conf
in the ConfigStore. Storing reload_conf
this way is equivalent to creating a file called reload_conf.yaml
that is discoverable by Hydra on the config search path. This reload_conf
node has a defaults list that loads both the schema base_config_cs
and config
. For this to work, the following two conditions must be met:
- the schema
base_config_cs
must be stored in the ConfigStore. This is accomplished by calling thestore_schema
function that we have imported fromapp.py
. - a config node with name specified by the variable
config_name
, i.e.config.yaml
in this example, must be discoverable by Hydra (which is taken care of here by callinginitialize_config_dir
).
Note that in foo.yaml
we have a defaults list ["base_config_cs", "_self_"]
that loads the schema base_config_cs
before loading the contents _self_
of foo
. In order for reload_conf
to reconstruct the app's config with the same merge order, base_config_cs
should come before config_name
in the defaults list belonging to reload_conf
.
The above approach could be taken one step further by removing the defaults list from foo.yaml
and using cs.store
to ensure the same defaults list is used in both the app and the reloading script
# app2.py
from dataclasses import dataclass
from enum import Enum
from typing import Any, List
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import MISSING, DictConfig
class SomeEnumClass(Enum):
ENUM1 = 1
ENUM2 = 2
@dataclass
class RootConfig:
defaults: List[Any] = MISSING
enum: SomeEnumClass = MISSING
x: int = 123
y: str = "abc"
def store_root_config(primary_config_name: str) -> None:
cs = ConfigStore.instance()
# defaults list defined here:
cs.store(
name="root_config", node=RootConfig(defaults=["_self_", primary_config_name])
)
@hydra.main(config_path=".", config_name="root_config")
def app(cfg: DictConfig) -> None:
print(cfg)
if __name__ == "__main__":
store_root_config("foo2")
app()
# foo2.yaml (note NO DEFAULTS LIST)
enum: ENUM1
x: 456
$ python app2.py hydra.job.chdir=false y=xyz
{'enum': , 'x': 456, 'y': 'xyz'}
# reload2.py
import os
from hydra import compose, initialize_config_dir
from hydra.core.config_store import ConfigStore
from app2 import store_root_config
config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-45-43")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
assert os.path.exists(f"{saved_cfg_dir}/{config_name}.yaml")
store_root_config("config")
with initialize_config_dir(config_dir=saved_cfg_dir):
cfg = compose("root_config")
print(cfg)
$ python reload2.py
{'enum': , 'x': 456, 'y': 'xyz'}
A simpler alternative approach is to use .hydra/overrides.yaml
to recompose the app's configuration based on the overrides that were originally passed to Hydra:
# reload3.py
import os
import yaml
from hydra import compose, initialize
from app import store_schema
config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-19-56")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
overrides_path = f"{saved_cfg_dir}/overrides.yaml"
assert os.path.exists(overrides_path)
overrides = yaml.unsafe_load(open(overrides_path, "r"))
print(f"{overrides=}")
store_schema()
with initialize(config_path="."):
cfg = compose("foo", overrides=overrides)
print(cfg)
$ python reload3.py
overrides=['y=xyz']
{'enum': , 'x': 456, 'y': 'xyz'}
This approach has its drawbacks: if your app's configuration involves some non-hermetic operation like querying a timestamp (e.g. via Hydra's now
resolver) or looking up an environment variable (e.g. via the oc.env
resolver), the configuration composed by reload.py
might be different from the original version loaded in app.py
.
QUESTION
I would like to use python datatypes - both built-in and imported from libraries such as numpy, tensorflow, etc - as arguments in my hydra configuration. Something like:
# config.yaml
arg1: np.float32
arg2: tf.float16
I'm currently doing this instead:
# config.yaml
arg1: 'float32'
arg2: 'float16
# my python code
# ...
DTYPES_LOOKUP = {
'float32': np.float32,
'float16': tf.float16
}
arg1 = DTYPES_LOOKUP[config.arg1]
arg2 = DTYPES_LOOKUP[config.arg2]
Is there a more hydronic/elegant solution? Thanks!
ANSWER
Answered 2022-Jan-19 at 22:13Does the hydra.utils.get_class
function solve this problem for you?
# config.yaml
arg1: numpy.float32 # note: use "numpy" here, not "np"
arg2: tensorflow.float16
# python code
...
from hydra.utils import get_class
arg1 = get_class(config.arg1)
arg2 = get_class(config.arg2)
Based on miccio's comment below, here is a demonstration using an OmegaConf custom resolver to wrap the get_class
function.
from omegaconf import OmegaConf
from hydra.utils import get_class
OmegaConf.register_new_resolver(name="get_cls", resolver=lambda cls: get_class(cls))
config = OmegaConf.create("""
# config.yaml
arg1: "${get_cls: numpy.float32}"
arg2: "${get_cls: tensorflow.float16}"
""")
arg1 = config.arg1
arg1 = config.arg2
It turns out that get_class("numpy.float32")
succeeds but get_class("tensorflow.float16")
raises a ValueError. The reason is that get_class
checks that the returned value is indeed a class (using isinstance(cls, type)
).
The function hydra.utils.get_method
is slightly more permissive, checking only that the returned value is a callable, but this still does not work with tf.float16
.
>>> isinstance(tf.float16, type)
False
>>> callable(tf.float16)
False
A custom resolver wrapping the tensorflow.as_dtype
function might be in order.
>>> tf.as_dtype("float16")
tf.float16
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install hydra
Run your own OAuth 2.0 Server - step by step guide: A in-depth look at setting up ORY Hydra and performing a variety of OAuth 2.0 Flows.
ORY Hydra 5 Minute Tutorial: Set up and use ORY Hydra using Docker Compose in under 5 Minutes. Good for quickly hacking a Proof of Concept.
Install and Set Up ORY Hydra: An advanced look at installation options and interaction with ORY Hydra.
Integrating your Login and Consent UI with ORY Hydra: The go-to place if you wish to adopt ORY Hydra in your new or existing stack.
What is ORY Hydra? Who's using it? OAuth2 and OpenID Connect: Open Standards! OpenID Connect Certified
Quickstart 5 minutes tutorial: Run your very own OAuth2 environment Installation
Ecosystem ORY Kratos: Identity and User Infrastructure and Management ORY Hydra: OAuth2 & OpenID Connect Server ORY Oathkeeper: Identity & Access Proxy ORY Keto: Access Control Policies as a Server
Security Disclosing vulnerabilities
Benchmarks
Telemetry
Documentation Guide HTTP API documentation Upgrading and Changelog Command line documentation Develop Dependencies Formatting Code Running Tests Short Tests Regular Tests E2E Tests Build Docker Run the Docker Compose quickstarts
Libraries and third-party projects
Blog posts & articles
This section is a starter guide to working with ORY Hydra. In-depth docs are available as well:.
The documentation is available here.
The REST API documentation is available here.
Head over to the ORY Developer Documentation to learn how to install ORY Hydra on Linux, macOS, Windows, and Docker and how to build ORY Hydra from source.
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kitsβ
Save this library and start creating your kit
Share this Page