archive-be.com » BE » B » BENNYMICHIELSEN.BE

Total: 348

Choose link from "Titles, links and description words view":

Or switch to "Titles and links view".
  • Benny Michielsen – .net developer with a touch of Cocoa
    partitions myFile Year Month Day txt It s only possible to use the partitionedBy section in the folder structure as shown above If you think this is a nice feature go vote here The price of the current setup is determined by a couple of things First we have a low frequency activity that s an activity that runs daily or less The first 5 are free so we have 25 activities remaining The pricing of an activity is determined on the place where it occurs on premise or in the cloud I m assuming here it s an on premise activity since the files are not located in Azure I ve asked around if this assumption is correct but don t have a response yet The pricing of an on premise activity is 0 5586 per activity So that would mean almost 14 for this daily snapshot each month If we modified everything to run hourly we d have to pay 554 80 per month You can find more info on the pricing on their website In this scenario I ve demonstrated how to get started with Azure Data Factory The real power however lies in the transformation steps which you can add Instead of doing a simple copy the data can be read combined and stored in many different forms A topic for a future post Upside Rich editor Fairly easy to get started No custom application to write On premise support via the Data Management Gateway No firewall settings need to be changed Downside Can get quite expensive Author BennyM Posted on 07 07 2015 07 07 2015 Categories Software Development Tags azure data factory 1 Comment on Partitioning and wildcards in an Azure Data Factory pipeline Copying files with Azure Data Factory The goal of Azure Data Factory is to create a pipeline which gathers a lot of data sources and produces a reliable source of information which can be used by other applications The pain of interfacing with every differnt type of datastore is abstracted away from every consuming application You can have relational databases flat files whatever and create a pipeline which transforms and enriches the data In an update published the end of March it was announced that you can also copy files I wanted to try this out and it proved to be a bit more cumbersome than I first imagined Let s take a look I want to create a very basic flow Let s say I have an application which is populating files in a folder and I now want to move the file into Azure blob storage I can use the same use case as mentioned in my previous post I m placing data leaks in a folder and need them to be sent online for further processing After creating a data factory in your Azure account we ll need the following components Two linked services A connection to my on premises folder A connection to my blob storage account Two datasets also called tables A dataset containing info on where my data is stored on premise and how many times per day it can be fetched A dataset which has info on how and where to store the data in blob storage One pipeline which contains an activity which connects the datasets The connection to the on premise file is handled by an application which you need to install on premise By navigating to the linked services slice you can add a data gateway To configure a gateway you only need to provide a name you can then download the data gateway application and install it on premise After installing the application you need to enter the key which can be viewed in the Azure portal As far as the on premise configuration you are done You do not need to configure any firewall ports but you can only install it once on a PC So far the wizards Now we need to create the pipeline After clicking on the Author and deploy tile The browser navigates to an online editor You create linked services datasets and pipelines by using JSON When clicking on any of the menu options you can select a template to get started As mentioned earlier I needed two linked services You can create those via the new data store option The first one I ll create is the on premise file data store name OnPremisesFileSystemLinkedService properties host localhost gatewayName mydatagateway userId BennyM password type OnPremisesFileSystemLinkedService The options you need to configure are host which will be the name of the machine which contains the files folder you need to connect to gatewayName which has to match the name of the gateway which we created earlier a userid and password or encryptedcredentials to use to connect from the gateway to the target machine type which needs to be OnPremisesFileSystemLinkedService name which we will use later The next one will be the connection to the Azure storage You can get the template by clicking New data store and selecting Azure storage name StorageLinkedService properties connectionString DefaultEndpointsProtocol https AccountName bennymdatafactory AccountKey type AzureStorageLinkedService The options you need to configure are name which we will use later connectionstring which needs to match your connectionstring for Azure storage type which needs to be AzureStorageLinkedService Next one the first dataset on premises file name OnPremisesFile properties location type OnPremisesFileSystemLocation folderPath c Temp linkedServiceName OnPremisesFileSystemLinkedService availability frequency Day interval 1 waitOnExternal dataDelay 00 10 00 retryInterval 00 01 00 retryTimeout 00 10 00 maximumRetry 3 The options you need to configure again a name which we ll use later type which has to be OnPremisesFileSystemLocation folderpath where is the folder I want to sync Note the double slashes linkedServiceName this has to be the same value which we used earlier when we created the data store for the on premises gateway availability how many times will the on premises file or folder by synchronized What s very important is the waitOnExternal You have to configure this if the data is not produced by the data factory itself In this case it s an external source so I have to fill in some values Our next dataset is the Azure blob name AzureBlobDatasetTemplate properties location type AzureBlobLocation folderPath myblobcontainer linkedServiceName StorageLinkedService availability frequency Day interval 1 Fairly easy to configure again a name the type which has to be AzureBlobLocation the folderPath wich will be the path inside my Azure blob storage account which was configured in the linked service linkedServiceName which has to match the name we used earlier Then the actual workflow the pipeline name CopyFileToBlobPipeline properties activities type CopyActivity transformation source type FileSystemSource sink type BlobSink writeBatchSize 0 writeBatchTimeout 00 00 00 inputs name OnPremisesFile outputs name AzureBlobDatasetTemplate policy timeout 00 05 00 concurrency 4 name Ingress start 2015 06 28T00 00 00Z end 2015 06 30T00 00 00Z I will not go over every property What s important is that in the copy activity we tie our two datasets together The start and end times indicate the time period our pipeline will be active This is very important as I ve not found any way you can pause a running pipeline If for instance I indicated earlier that I want my folder to be copied every 15 minutes and indicate a start date in the pipline in the year 2010 the pipeline will do all historical runs So be careful when you configure this as Azure Data Factory isn t the cheapest service around Every time you create a linked service dataset or pipeline you click deploy The editor will then validate if all the names you mention are actually correct The editor is quite good as it even detects errors in datatypes and gives you help and intellisense control space And with all this deployed the folder is being synchronized online This is probably one of the most expensive file copy samples In a next post I ll investigate more features One of the drawbacks of the current setup is that every time the sync runs all files are overwritten All files are copied all the time as is Sources Enable your pipelines to work with on premises data Data Factory JSON Scripting Reference Author BennyM Posted on 30 06 2015 01 07 2015 Categories Software Development Tags azure data factory Leave a comment on Copying files with Azure Data Factory Using HDInsight Storm to process 100 million events In my last post I threw 100 randomly generated email addresses to an Azure event hub and it didn t even budge Now it s time to process the data and store it in a setup that resembles haveibeenpwned With the events now in the event hub I could create another cloud service which uses the NET SDK to read the data There are two ways to implement this either with the EventHubReceiver class or with the EventProcessorHost I ll try these out in future post For now I wanted to use something else HDInsight Storm HDInsight is Hadoop as PaaS Storm is a computation system to process streams of data and as advertised on their website should be able to process millions of tuples per second Sounds like a good way to handle the 100 million e mail addresses I have waiting In order to use Storm you need to understand only a few concepts Tuple a piece of information that needs to be processed Spout reads from a stream and returns tuples Bolt processes tuples and can forward or produce new tuples Topology links spouts and bolts together This is a very rudimentary explanation but should be enough for this post In order to write and publish a topology in Visual Studio you should have the Azure SDK and the HDInsight tools for Visual Studio Let s look at the different components we need The spout will be reading from our event hub Microsoft has already written a spout which you can use However it is written in Java Storm is not bound to one language a core feature and with HDInsight you can have hybrid topologies Any work you have already invested in Storm can be reused Java and C spouts and bolts can even work together Let s look at the configuration that we need to get our spout up and running TopologyBuilder topologyBuilder new TopologyBuilder EmailProcessingTopology int partitionCount Properties Settings Default EventHubPartitionCount JavaComponentConstructor constructor JavaComponentConstructor CreateFromClojureExpr String Format com microsoft eventhubs spout EventHubSpout com microsoft eventhubs spout EventHubSpoutConfig 0 1 2 3 4 5 Properties Settings Default EventHubPolicyName Properties Settings Default EventHubPolicyKey Properties Settings Default EventHubNamespace Properties Settings Default EventHubName partitionCount topologyBuilder SetJavaSpout EventHubSpout constructor partitionCount This code can be found in the samples of the HDInsight team and is pretty straightforward Create an eventhubspout and add it to the topology The partitionCount indicates how many executors and tasks there should be and i t s suggested that this should match the amount of partitions of your event hub It gets more interesting in the first bolt A bolt is a class that implements ISCPBolt This interface has one method Execute which receives a tuple public void Execute SCPTuple tuple string emailAddress string tuple GetValue 0 if string IsNullOrWhiteSpace emailAddress JObject eventData JObject Parse emailAddress try var address new System Net Mail MailAddress string eventData address var leakedIn string eventData leak var breachedAccount new BreachedAccount address leakedIn var retrieveOperation TableOperation Retrieve breachedAccount PartitionKey breachedAccount RowKey var retrievedResult table Execute retrieveOperation var existingAccount Device retrievedResult Result if existingAccount null TableOperation insertOperation TableOperation Insert new BreachedAccount address leakedIn table Execute insertOperation ctx Emit Constants DEFAULT STREAM ID new tuple new Values address Address else existingAccount BreachedIn breachedAccount BreachedIn TableOperation insertOperation TableOperation Replace existingAccount table Execute insertOperation this ctx Ack tuple catch Exception ex Context Logger Error eventData ToString Context Logger Error ex Message this ctx Fail tuple else Context Logger Info empty address this ctx Ack tuple The tuple is the JSON object which was send to the Azure event hub in my previous post It contains the email address and the website which was breached Table storage is queried to see if the email address is already present if this is the case the entry is updated otherwise the new account is added to the system Only new accounts are emitted for further processing The second bolt in the system will insert new accounts into a SQL database Its execute method is quite different public void Execute SCPTuple tuple Check if the tuple is of type Tick Tuple if IsTickTuple tuple if DateTime UtcNow lastRun TimeSpan FromSeconds batchIntervalInSec Context Logger Info Time to purge FinishBatch else breachedAccounts Add tuple if breachedAccounts Count batchSize Context Logger Info Max reached time to purge FinishBatch else Context Logger Info Not yet time to purge This bolt does not process every tuple it receives immediately Instead it keeps a list of tuples ready for processing when either a maximum count is reached or a specific amount of time has passed since the last purge In the FinishBatch method a SqlBulk copy is performed to insert records in a SQL database private void FinishBatch lastRun DateTime UtcNow DataTable table new DataTable table Columns Add Email if breachedAccounts Any foreach var emailAddress in breachedAccounts var row table NewRow row Email emailAddress GetString 0 table Rows Add row try using SqlBulkCopy bulkCopy new SqlBulkCopy connection bulkCopy DestinationTableName Emails bulkCopy ColumnMappings Add Email Email bulkCopy WriteToServer table foreach var emailAddress in breachedAccounts ctx Ack emailAddress catch Exception ex foreach var emailAddress in breachedAccounts ctx Fail emailAddress Context Logger Error ex Message finally breachedAccounts Clear Both the table storage bolt and the SQL bulk copy bolt have lines with ctx Ack Fail or Emit These are places where you can indicate whether processing of a tuple has succeeded failed or you are emitting a new tuple to be processed further downstream It also enables to replay tuples If the SQL bulk copy fails the tuple will be processed again at a later time The azure table spout expects your bolts to ack topologyBuilder SetBolt TableStorageBolt TableStorageBolt Get new Dictionary string List Constants DEFAULT STREAM ID new List address 256 true DeclareCustomizedJavaSerializer javaSerializerInfo shuffleGrouping EventHubSpout topologyBuilder SetBolt SQLBolt SQLStorageBolt Get new Dictionary string List 2 true shuffleGrouping TableStorageBolt Constants DEFAULT STREAM ID addConfigurations new Dictionary string string topology tick tuple freq secs 1 topologyBuilder SetTopologyConfig new Dictionary string string topology workers partitionCount ToString The two bolts also need to be added to the topology The tableStorageBolt is configured with a name a method which can be used to create instances and a description of its output In this case I m emitting the email address to the default stream The parallelism hint is also configured and the boolean flag which indicates this bolt supports acking Since this bolt sits behind the Java spout we also need to configure how the data of the spout can be deserialized The bolt is also configured to use shuffleGrouping meaning that the load will be spread evenly among all instances The SQLBolt is configured accordingly with the additional configuration to support time ticks After I created an HDInsight Storm cluster on Azure I was able to publish the topology from within Visual Studio The cluster itself was created in 15 minutes It took quite some time to find the right parallelism for this topology but with the code I ve posted I got these results after one hour Table storage is slow the 2 SQL bolts are able to keep up with the 256 table storage bolts After one hour my topology was able to process only 10 million records Even though I was able to send all 100 million events to the event hub in under two hours processing would take 5 times longer One of the causes for this is the test data The partition and row key for the table storage is based on the email address class BreachedAccount TableEntity public BreachedAccount public BreachedAccount System Net Mail MailAddress emailAddress string leakedIn this PartitionKey emailAddress Host this RowKey emailAddress Address public string BreachedIn get set Yet none of the email addresses in my test data share the same partition key If the data was more realistic I would be able to use batches In the original Adobe data breach 115 million addresses were part of 130 different domains Table storage allows you to send batches of 100 records to the same partition key which should result in less processing time I ll regenerate the test data with the same characteristics modify the table storage bolt and rerun the experiment at a later time hoping to get a better result The original standard instance of SQL Server was scaled to an S3 which allowed me to stay just below the max throughput with batches of 1000 records At the moment I actually have the same performance as Troy s setup but with quite a different approach Storm and HDInsight were new to me and I enjoyed the programming model The scale and possibilities it can give are enormous yet every part of your topology needs quite some attention if you want to have a high throughput solution The NET part of the SDK is not yet entirely complete so you can t just port Java snippets to C and expect them to work I also ran into some NotImplementedExceptions Microsoft is still busy implementing the SDK and the team is actively adding examples and functionality Further reading Event Hubs Programming Guide Storm Applied Introduction to Apache Storm on HDInsight Real time analytics for Hadoop HDInsight Storm Examples Process events from Azure Event Hubs with Storm on HDInsight Performance Tuning for HDInsight Storm and Microsoft Azure EventHub Understanding the Parallelism of a Storm Topology Author BennyM Posted on 08 06 2015 08 06 2015 Categories Software Development Tags azure event hub hdinsight storm 4 Comments on Using HDInsight Storm to process 100 million events Data leak processing

    Original URL path: http://blog.bennymichielsen.be/ (2016-04-29)
    Open archived version from archive


  • Getting up and running with Mono and Raspberry Pi 3 – Benny Michielsen
    had I could program in C Python C and many others But time was limited this weekend I live and breath NET so I changed the list to NET Core or Mono I chose Mono because I had experimented with it years ago and NET Core has not yet reached a stable point Toying around with alpha and beta releases was not on my todo list for today The default package repository has a very old version of Mono so you need to follow the instructions on the Mono site Add the signing key and package repository to your system then run sudo apt get install mono runtime sudo apt key adv keyserver hkp keyserver ubuntu com 80 recv keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo deb http download mono project com repo debian wheezy main sudo tee etc apt sources list d mono xamarin list sudo apt get update sudo apt get install mono runtime I created a Hello World console application on my Windows 10 laptop and used PSFTP to copy the exe to the Pi It just worked Then the search was on to find a library to interface with the GPIO pins on the Pi After looking around I found Raspberry Sharp IO It had the API I wanted You can use the event model to track changes in the GPIO pins just what I needed var pin2Sensor ConnectorPin P1Pin11 Input GpioConnection connection new GpioConnection pin2Sensor connection PinStatusChanged sender statusArgs Console WriteLine Pin changed statusArgs Configuration Name Deploying this to the Pi however resulted in catastrophic failure Some weird error message pi raspberrypi ticktack sudo mono TickTackConsole exe Missing method ctor in assembly home pi ticktack Raspberry IO GeneralPurpose dll type System Runtime CompilerServices ExtensionAttribute Can t find custom attr constructor image home pi ticktack Raspberry IO GeneralPurpose dll mtoken 0x0a000014 Assertion at class c 5597 condition mono loader get last error not met Stacktrace Native stacktrace Debug info from gdb New LWP 1965 Thread debugging using libthread db enabled Using host libthread db library lib arm linux gnueabihf libthread db so 1 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 40 sysdeps unix sysv linux waitpid c No such file or directory Id Target Id Frame 2 Thread 0x769f3430 LWP 1965 mono 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 Thread 0x76f5e000 LWP 1961 mono 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 Thread 2 Thread 0x769f3430 LWP 1965 0 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 0x76e65af4 in new sem wait sem 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 69 2 0x00219f98 in mono sem wait 3 0x0019091c in Backtrace stopped previous frame identical to this

    Original URL path: http://blog.bennymichielsen.be/2016/03/14/getting-up-and-running-with-mono-and-raspberry-pi-3/ (2016-04-29)
    Open archived version from archive

  • BennyM – Benny Michielsen
    dynamic partitions myFile Year Month Day txt It s only possible to use the partitionedBy section in the folder structure as shown above If you think this is a nice feature go vote here The price of the current setup is determined by a couple of things First we have a low frequency activity that s an activity that runs daily or less The first 5 are free so we have 25 activities remaining The pricing of an activity is determined on the place where it occurs on premise or in the cloud I m assuming here it s an on premise activity since the files are not located in Azure I ve asked around if this assumption is correct but don t have a response yet The pricing of an on premise activity is 0 5586 per activity So that would mean almost 14 for this daily snapshot each month If we modified everything to run hourly we d have to pay 554 80 per month You can find more info on the pricing on their website In this scenario I ve demonstrated how to get started with Azure Data Factory The real power however lies in the transformation steps which you can add Instead of doing a simple copy the data can be read combined and stored in many different forms A topic for a future post Upside Rich editor Fairly easy to get started No custom application to write On premise support via the Data Management Gateway No firewall settings need to be changed Downside Can get quite expensive Author BennyM Posted on 07 07 2015 07 07 2015 Categories Software Development Tags azure data factory 1 Comment on Partitioning and wildcards in an Azure Data Factory pipeline Copying files with Azure Data Factory The goal of Azure Data Factory is to create a pipeline which gathers a lot of data sources and produces a reliable source of information which can be used by other applications The pain of interfacing with every differnt type of datastore is abstracted away from every consuming application You can have relational databases flat files whatever and create a pipeline which transforms and enriches the data In an update published the end of March it was announced that you can also copy files I wanted to try this out and it proved to be a bit more cumbersome than I first imagined Let s take a look I want to create a very basic flow Let s say I have an application which is populating files in a folder and I now want to move the file into Azure blob storage I can use the same use case as mentioned in my previous post I m placing data leaks in a folder and need them to be sent online for further processing After creating a data factory in your Azure account we ll need the following components Two linked services A connection to my on premises folder A connection to my blob storage account Two datasets also called tables A dataset containing info on where my data is stored on premise and how many times per day it can be fetched A dataset which has info on how and where to store the data in blob storage One pipeline which contains an activity which connects the datasets The connection to the on premise file is handled by an application which you need to install on premise By navigating to the linked services slice you can add a data gateway To configure a gateway you only need to provide a name you can then download the data gateway application and install it on premise After installing the application you need to enter the key which can be viewed in the Azure portal As far as the on premise configuration you are done You do not need to configure any firewall ports but you can only install it once on a PC So far the wizards Now we need to create the pipeline After clicking on the Author and deploy tile The browser navigates to an online editor You create linked services datasets and pipelines by using JSON When clicking on any of the menu options you can select a template to get started As mentioned earlier I needed two linked services You can create those via the new data store option The first one I ll create is the on premise file data store name OnPremisesFileSystemLinkedService properties host localhost gatewayName mydatagateway userId BennyM password type OnPremisesFileSystemLinkedService The options you need to configure are host which will be the name of the machine which contains the files folder you need to connect to gatewayName which has to match the name of the gateway which we created earlier a userid and password or encryptedcredentials to use to connect from the gateway to the target machine type which needs to be OnPremisesFileSystemLinkedService name which we will use later The next one will be the connection to the Azure storage You can get the template by clicking New data store and selecting Azure storage name StorageLinkedService properties connectionString DefaultEndpointsProtocol https AccountName bennymdatafactory AccountKey type AzureStorageLinkedService The options you need to configure are name which we will use later connectionstring which needs to match your connectionstring for Azure storage type which needs to be AzureStorageLinkedService Next one the first dataset on premises file name OnPremisesFile properties location type OnPremisesFileSystemLocation folderPath c Temp linkedServiceName OnPremisesFileSystemLinkedService availability frequency Day interval 1 waitOnExternal dataDelay 00 10 00 retryInterval 00 01 00 retryTimeout 00 10 00 maximumRetry 3 The options you need to configure again a name which we ll use later type which has to be OnPremisesFileSystemLocation folderpath where is the folder I want to sync Note the double slashes linkedServiceName this has to be the same value which we used earlier when we created the data store for the on premises gateway availability how many times will the on premises file or folder by synchronized What s very important is the waitOnExternal You have to configure this if the data is not produced by the data factory itself In this case it s an external source so I have to fill in some values Our next dataset is the Azure blob name AzureBlobDatasetTemplate properties location type AzureBlobLocation folderPath myblobcontainer linkedServiceName StorageLinkedService availability frequency Day interval 1 Fairly easy to configure again a name the type which has to be AzureBlobLocation the folderPath wich will be the path inside my Azure blob storage account which was configured in the linked service linkedServiceName which has to match the name we used earlier Then the actual workflow the pipeline name CopyFileToBlobPipeline properties activities type CopyActivity transformation source type FileSystemSource sink type BlobSink writeBatchSize 0 writeBatchTimeout 00 00 00 inputs name OnPremisesFile outputs name AzureBlobDatasetTemplate policy timeout 00 05 00 concurrency 4 name Ingress start 2015 06 28T00 00 00Z end 2015 06 30T00 00 00Z I will not go over every property What s important is that in the copy activity we tie our two datasets together The start and end times indicate the time period our pipeline will be active This is very important as I ve not found any way you can pause a running pipeline If for instance I indicated earlier that I want my folder to be copied every 15 minutes and indicate a start date in the pipline in the year 2010 the pipeline will do all historical runs So be careful when you configure this as Azure Data Factory isn t the cheapest service around Every time you create a linked service dataset or pipeline you click deploy The editor will then validate if all the names you mention are actually correct The editor is quite good as it even detects errors in datatypes and gives you help and intellisense control space And with all this deployed the folder is being synchronized online This is probably one of the most expensive file copy samples In a next post I ll investigate more features One of the drawbacks of the current setup is that every time the sync runs all files are overwritten All files are copied all the time as is Sources Enable your pipelines to work with on premises data Data Factory JSON Scripting Reference Author BennyM Posted on 30 06 2015 01 07 2015 Categories Software Development Tags azure data factory Leave a comment on Copying files with Azure Data Factory Using HDInsight Storm to process 100 million events In my last post I threw 100 randomly generated email addresses to an Azure event hub and it didn t even budge Now it s time to process the data and store it in a setup that resembles haveibeenpwned With the events now in the event hub I could create another cloud service which uses the NET SDK to read the data There are two ways to implement this either with the EventHubReceiver class or with the EventProcessorHost I ll try these out in future post For now I wanted to use something else HDInsight Storm HDInsight is Hadoop as PaaS Storm is a computation system to process streams of data and as advertised on their website should be able to process millions of tuples per second Sounds like a good way to handle the 100 million e mail addresses I have waiting In order to use Storm you need to understand only a few concepts Tuple a piece of information that needs to be processed Spout reads from a stream and returns tuples Bolt processes tuples and can forward or produce new tuples Topology links spouts and bolts together This is a very rudimentary explanation but should be enough for this post In order to write and publish a topology in Visual Studio you should have the Azure SDK and the HDInsight tools for Visual Studio Let s look at the different components we need The spout will be reading from our event hub Microsoft has already written a spout which you can use However it is written in Java Storm is not bound to one language a core feature and with HDInsight you can have hybrid topologies Any work you have already invested in Storm can be reused Java and C spouts and bolts can even work together Let s look at the configuration that we need to get our spout up and running TopologyBuilder topologyBuilder new TopologyBuilder EmailProcessingTopology int partitionCount Properties Settings Default EventHubPartitionCount JavaComponentConstructor constructor JavaComponentConstructor CreateFromClojureExpr String Format com microsoft eventhubs spout EventHubSpout com microsoft eventhubs spout EventHubSpoutConfig 0 1 2 3 4 5 Properties Settings Default EventHubPolicyName Properties Settings Default EventHubPolicyKey Properties Settings Default EventHubNamespace Properties Settings Default EventHubName partitionCount topologyBuilder SetJavaSpout EventHubSpout constructor partitionCount This code can be found in the samples of the HDInsight team and is pretty straightforward Create an eventhubspout and add it to the topology The partitionCount indicates how many executors and tasks there should be and i t s suggested that this should match the amount of partitions of your event hub It gets more interesting in the first bolt A bolt is a class that implements ISCPBolt This interface has one method Execute which receives a tuple public void Execute SCPTuple tuple string emailAddress string tuple GetValue 0 if string IsNullOrWhiteSpace emailAddress JObject eventData JObject Parse emailAddress try var address new System Net Mail MailAddress string eventData address var leakedIn string eventData leak var breachedAccount new BreachedAccount address leakedIn var retrieveOperation TableOperation Retrieve breachedAccount PartitionKey breachedAccount RowKey var retrievedResult table Execute retrieveOperation var existingAccount Device retrievedResult Result if existingAccount null TableOperation insertOperation TableOperation Insert new BreachedAccount address leakedIn table Execute insertOperation ctx Emit Constants DEFAULT STREAM ID new tuple new Values address Address else existingAccount BreachedIn breachedAccount BreachedIn TableOperation insertOperation TableOperation Replace existingAccount table Execute insertOperation this ctx Ack tuple catch Exception ex Context Logger Error eventData ToString Context Logger Error ex Message this ctx Fail tuple else Context Logger Info empty address this ctx Ack tuple The tuple is the JSON object which was send to the Azure event hub in my previous post It contains the email address and the website which was breached Table storage is queried to see if the email address is already present if this is the case the entry is updated otherwise the new account is added to the system Only new accounts are emitted for further processing The second bolt in the system will insert new accounts into a SQL database Its execute method is quite different public void Execute SCPTuple tuple Check if the tuple is of type Tick Tuple if IsTickTuple tuple if DateTime UtcNow lastRun TimeSpan FromSeconds batchIntervalInSec Context Logger Info Time to purge FinishBatch else breachedAccounts Add tuple if breachedAccounts Count batchSize Context Logger Info Max reached time to purge FinishBatch else Context Logger Info Not yet time to purge This bolt does not process every tuple it receives immediately Instead it keeps a list of tuples ready for processing when either a maximum count is reached or a specific amount of time has passed since the last purge In the FinishBatch method a SqlBulk copy is performed to insert records in a SQL database private void FinishBatch lastRun DateTime UtcNow DataTable table new DataTable table Columns Add Email if breachedAccounts Any foreach var emailAddress in breachedAccounts var row table NewRow row Email emailAddress GetString 0 table Rows Add row try using SqlBulkCopy bulkCopy new SqlBulkCopy connection bulkCopy DestinationTableName Emails bulkCopy ColumnMappings Add Email Email bulkCopy WriteToServer table foreach var emailAddress in breachedAccounts ctx Ack emailAddress catch Exception ex foreach var emailAddress in breachedAccounts ctx Fail emailAddress Context Logger Error ex Message finally breachedAccounts Clear Both the table storage bolt and the SQL bulk copy bolt have lines with ctx Ack Fail or Emit These are places where you can indicate whether processing of a tuple has succeeded failed or you are emitting a new tuple to be processed further downstream It also enables to replay tuples If the SQL bulk copy fails the tuple will be processed again at a later time The azure table spout expects your bolts to ack topologyBuilder SetBolt TableStorageBolt TableStorageBolt Get new Dictionary string List Constants DEFAULT STREAM ID new List address 256 true DeclareCustomizedJavaSerializer javaSerializerInfo shuffleGrouping EventHubSpout topologyBuilder SetBolt SQLBolt SQLStorageBolt Get new Dictionary string List 2 true shuffleGrouping TableStorageBolt Constants DEFAULT STREAM ID addConfigurations new Dictionary string string topology tick tuple freq secs 1 topologyBuilder SetTopologyConfig new Dictionary string string topology workers partitionCount ToString The two bolts also need to be added to the topology The tableStorageBolt is configured with a name a method which can be used to create instances and a description of its output In this case I m emitting the email address to the default stream The parallelism hint is also configured and the boolean flag which indicates this bolt supports acking Since this bolt sits behind the Java spout we also need to configure how the data of the spout can be deserialized The bolt is also configured to use shuffleGrouping meaning that the load will be spread evenly among all instances The SQLBolt is configured accordingly with the additional configuration to support time ticks After I created an HDInsight Storm cluster on Azure I was able to publish the topology from within Visual Studio The cluster itself was created in 15 minutes It took quite some time to find the right parallelism for this topology but with the code I ve posted I got these results after one hour Table storage is slow the 2 SQL bolts are able to keep up with the 256 table storage bolts After one hour my topology was able to process only 10 million records Even though I was able to send all 100 million events to the event hub in under two hours processing would take 5 times longer One of the causes for this is the test data The partition and row key for the table storage is based on the email address class BreachedAccount TableEntity public BreachedAccount public BreachedAccount System Net Mail MailAddress emailAddress string leakedIn this PartitionKey emailAddress Host this RowKey emailAddress Address public string BreachedIn get set Yet none of the email addresses in my test data share the same partition key If the data was more realistic I would be able to use batches In the original Adobe data breach 115 million addresses were part of 130 different domains Table storage allows you to send batches of 100 records to the same partition key which should result in less processing time I ll regenerate the test data with the same characteristics modify the table storage bolt and rerun the experiment at a later time hoping to get a better result The original standard instance of SQL Server was scaled to an S3 which allowed me to stay just below the max throughput with batches of 1000 records At the moment I actually have the same performance as Troy s setup but with quite a different approach Storm and HDInsight were new to me and I enjoyed the programming model The scale and possibilities it can give are enormous yet every part of your topology needs quite some attention if you want to have a high throughput solution The NET part of the SDK is not yet entirely complete so you can t just port Java snippets to C and expect them to work I also ran into some NotImplementedExceptions Microsoft is still busy implementing the SDK and the team is actively adding examples and functionality Further reading Event Hubs Programming Guide Storm Applied Introduction to Apache Storm on HDInsight Real time analytics for Hadoop HDInsight Storm Examples Process events from Azure Event Hubs with Storm on HDInsight Performance Tuning for HDInsight Storm and Microsoft Azure EventHub Understanding the Parallelism of a Storm Topology Author BennyM Posted on 08 06 2015 08 06 2015 Categories Software Development Tags azure event hub hdinsight storm 4 Comments on Using HDInsight Storm to process 100 million events Data leak

    Original URL path: http://blog.bennymichielsen.be/author/bennym/ (2016-04-29)
    Open archived version from archive

  • Software Development – Benny Michielsen
    with the dynamic partitions myFile Year Month Day txt It s only possible to use the partitionedBy section in the folder structure as shown above If you think this is a nice feature go vote here The price of the current setup is determined by a couple of things First we have a low frequency activity that s an activity that runs daily or less The first 5 are free so we have 25 activities remaining The pricing of an activity is determined on the place where it occurs on premise or in the cloud I m assuming here it s an on premise activity since the files are not located in Azure I ve asked around if this assumption is correct but don t have a response yet The pricing of an on premise activity is 0 5586 per activity So that would mean almost 14 for this daily snapshot each month If we modified everything to run hourly we d have to pay 554 80 per month You can find more info on the pricing on their website In this scenario I ve demonstrated how to get started with Azure Data Factory The real power however lies in the transformation steps which you can add Instead of doing a simple copy the data can be read combined and stored in many different forms A topic for a future post Upside Rich editor Fairly easy to get started No custom application to write On premise support via the Data Management Gateway No firewall settings need to be changed Downside Can get quite expensive Author BennyM Posted on 07 07 2015 07 07 2015 Categories Software Development Tags azure data factory 1 Comment on Partitioning and wildcards in an Azure Data Factory pipeline Copying files with Azure Data Factory The goal of Azure Data Factory is to create a pipeline which gathers a lot of data sources and produces a reliable source of information which can be used by other applications The pain of interfacing with every differnt type of datastore is abstracted away from every consuming application You can have relational databases flat files whatever and create a pipeline which transforms and enriches the data In an update published the end of March it was announced that you can also copy files I wanted to try this out and it proved to be a bit more cumbersome than I first imagined Let s take a look I want to create a very basic flow Let s say I have an application which is populating files in a folder and I now want to move the file into Azure blob storage I can use the same use case as mentioned in my previous post I m placing data leaks in a folder and need them to be sent online for further processing After creating a data factory in your Azure account we ll need the following components Two linked services A connection to my on premises folder A connection to my blob storage account Two datasets also called tables A dataset containing info on where my data is stored on premise and how many times per day it can be fetched A dataset which has info on how and where to store the data in blob storage One pipeline which contains an activity which connects the datasets The connection to the on premise file is handled by an application which you need to install on premise By navigating to the linked services slice you can add a data gateway To configure a gateway you only need to provide a name you can then download the data gateway application and install it on premise After installing the application you need to enter the key which can be viewed in the Azure portal As far as the on premise configuration you are done You do not need to configure any firewall ports but you can only install it once on a PC So far the wizards Now we need to create the pipeline After clicking on the Author and deploy tile The browser navigates to an online editor You create linked services datasets and pipelines by using JSON When clicking on any of the menu options you can select a template to get started As mentioned earlier I needed two linked services You can create those via the new data store option The first one I ll create is the on premise file data store name OnPremisesFileSystemLinkedService properties host localhost gatewayName mydatagateway userId BennyM password type OnPremisesFileSystemLinkedService The options you need to configure are host which will be the name of the machine which contains the files folder you need to connect to gatewayName which has to match the name of the gateway which we created earlier a userid and password or encryptedcredentials to use to connect from the gateway to the target machine type which needs to be OnPremisesFileSystemLinkedService name which we will use later The next one will be the connection to the Azure storage You can get the template by clicking New data store and selecting Azure storage name StorageLinkedService properties connectionString DefaultEndpointsProtocol https AccountName bennymdatafactory AccountKey type AzureStorageLinkedService The options you need to configure are name which we will use later connectionstring which needs to match your connectionstring for Azure storage type which needs to be AzureStorageLinkedService Next one the first dataset on premises file name OnPremisesFile properties location type OnPremisesFileSystemLocation folderPath c Temp linkedServiceName OnPremisesFileSystemLinkedService availability frequency Day interval 1 waitOnExternal dataDelay 00 10 00 retryInterval 00 01 00 retryTimeout 00 10 00 maximumRetry 3 The options you need to configure again a name which we ll use later type which has to be OnPremisesFileSystemLocation folderpath where is the folder I want to sync Note the double slashes linkedServiceName this has to be the same value which we used earlier when we created the data store for the on premises gateway availability how many times will the on premises file or folder by synchronized What s very important is the waitOnExternal You have to configure this if the data is not produced by the data factory itself In this case it s an external source so I have to fill in some values Our next dataset is the Azure blob name AzureBlobDatasetTemplate properties location type AzureBlobLocation folderPath myblobcontainer linkedServiceName StorageLinkedService availability frequency Day interval 1 Fairly easy to configure again a name the type which has to be AzureBlobLocation the folderPath wich will be the path inside my Azure blob storage account which was configured in the linked service linkedServiceName which has to match the name we used earlier Then the actual workflow the pipeline name CopyFileToBlobPipeline properties activities type CopyActivity transformation source type FileSystemSource sink type BlobSink writeBatchSize 0 writeBatchTimeout 00 00 00 inputs name OnPremisesFile outputs name AzureBlobDatasetTemplate policy timeout 00 05 00 concurrency 4 name Ingress start 2015 06 28T00 00 00Z end 2015 06 30T00 00 00Z I will not go over every property What s important is that in the copy activity we tie our two datasets together The start and end times indicate the time period our pipeline will be active This is very important as I ve not found any way you can pause a running pipeline If for instance I indicated earlier that I want my folder to be copied every 15 minutes and indicate a start date in the pipline in the year 2010 the pipeline will do all historical runs So be careful when you configure this as Azure Data Factory isn t the cheapest service around Every time you create a linked service dataset or pipeline you click deploy The editor will then validate if all the names you mention are actually correct The editor is quite good as it even detects errors in datatypes and gives you help and intellisense control space And with all this deployed the folder is being synchronized online This is probably one of the most expensive file copy samples In a next post I ll investigate more features One of the drawbacks of the current setup is that every time the sync runs all files are overwritten All files are copied all the time as is Sources Enable your pipelines to work with on premises data Data Factory JSON Scripting Reference Author BennyM Posted on 30 06 2015 01 07 2015 Categories Software Development Tags azure data factory Leave a comment on Copying files with Azure Data Factory Using HDInsight Storm to process 100 million events In my last post I threw 100 randomly generated email addresses to an Azure event hub and it didn t even budge Now it s time to process the data and store it in a setup that resembles haveibeenpwned With the events now in the event hub I could create another cloud service which uses the NET SDK to read the data There are two ways to implement this either with the EventHubReceiver class or with the EventProcessorHost I ll try these out in future post For now I wanted to use something else HDInsight Storm HDInsight is Hadoop as PaaS Storm is a computation system to process streams of data and as advertised on their website should be able to process millions of tuples per second Sounds like a good way to handle the 100 million e mail addresses I have waiting In order to use Storm you need to understand only a few concepts Tuple a piece of information that needs to be processed Spout reads from a stream and returns tuples Bolt processes tuples and can forward or produce new tuples Topology links spouts and bolts together This is a very rudimentary explanation but should be enough for this post In order to write and publish a topology in Visual Studio you should have the Azure SDK and the HDInsight tools for Visual Studio Let s look at the different components we need The spout will be reading from our event hub Microsoft has already written a spout which you can use However it is written in Java Storm is not bound to one language a core feature and with HDInsight you can have hybrid topologies Any work you have already invested in Storm can be reused Java and C spouts and bolts can even work together Let s look at the configuration that we need to get our spout up and running TopologyBuilder topologyBuilder new TopologyBuilder EmailProcessingTopology int partitionCount Properties Settings Default EventHubPartitionCount JavaComponentConstructor constructor JavaComponentConstructor CreateFromClojureExpr String Format com microsoft eventhubs spout EventHubSpout com microsoft eventhubs spout EventHubSpoutConfig 0 1 2 3 4 5 Properties Settings Default EventHubPolicyName Properties Settings Default EventHubPolicyKey Properties Settings Default EventHubNamespace Properties Settings Default EventHubName partitionCount topologyBuilder SetJavaSpout EventHubSpout constructor partitionCount This code can be found in the samples of the HDInsight team and is pretty straightforward Create an eventhubspout and add it to the topology The partitionCount indicates how many executors and tasks there should be and i t s suggested that this should match the amount of partitions of your event hub It gets more interesting in the first bolt A bolt is a class that implements ISCPBolt This interface has one method Execute which receives a tuple public void Execute SCPTuple tuple string emailAddress string tuple GetValue 0 if string IsNullOrWhiteSpace emailAddress JObject eventData JObject Parse emailAddress try var address new System Net Mail MailAddress string eventData address var leakedIn string eventData leak var breachedAccount new BreachedAccount address leakedIn var retrieveOperation TableOperation Retrieve breachedAccount PartitionKey breachedAccount RowKey var retrievedResult table Execute retrieveOperation var existingAccount Device retrievedResult Result if existingAccount null TableOperation insertOperation TableOperation Insert new BreachedAccount address leakedIn table Execute insertOperation ctx Emit Constants DEFAULT STREAM ID new tuple new Values address Address else existingAccount BreachedIn breachedAccount BreachedIn TableOperation insertOperation TableOperation Replace existingAccount table Execute insertOperation this ctx Ack tuple catch Exception ex Context Logger Error eventData ToString Context Logger Error ex Message this ctx Fail tuple else Context Logger Info empty address this ctx Ack tuple The tuple is the JSON object which was send to the Azure event hub in my previous post It contains the email address and the website which was breached Table storage is queried to see if the email address is already present if this is the case the entry is updated otherwise the new account is added to the system Only new accounts are emitted for further processing The second bolt in the system will insert new accounts into a SQL database Its execute method is quite different public void Execute SCPTuple tuple Check if the tuple is of type Tick Tuple if IsTickTuple tuple if DateTime UtcNow lastRun TimeSpan FromSeconds batchIntervalInSec Context Logger Info Time to purge FinishBatch else breachedAccounts Add tuple if breachedAccounts Count batchSize Context Logger Info Max reached time to purge FinishBatch else Context Logger Info Not yet time to purge This bolt does not process every tuple it receives immediately Instead it keeps a list of tuples ready for processing when either a maximum count is reached or a specific amount of time has passed since the last purge In the FinishBatch method a SqlBulk copy is performed to insert records in a SQL database private void FinishBatch lastRun DateTime UtcNow DataTable table new DataTable table Columns Add Email if breachedAccounts Any foreach var emailAddress in breachedAccounts var row table NewRow row Email emailAddress GetString 0 table Rows Add row try using SqlBulkCopy bulkCopy new SqlBulkCopy connection bulkCopy DestinationTableName Emails bulkCopy ColumnMappings Add Email Email bulkCopy WriteToServer table foreach var emailAddress in breachedAccounts ctx Ack emailAddress catch Exception ex foreach var emailAddress in breachedAccounts ctx Fail emailAddress Context Logger Error ex Message finally breachedAccounts Clear Both the table storage bolt and the SQL bulk copy bolt have lines with ctx Ack Fail or Emit These are places where you can indicate whether processing of a tuple has succeeded failed or you are emitting a new tuple to be processed further downstream It also enables to replay tuples If the SQL bulk copy fails the tuple will be processed again at a later time The azure table spout expects your bolts to ack topologyBuilder SetBolt TableStorageBolt TableStorageBolt Get new Dictionary string List Constants DEFAULT STREAM ID new List address 256 true DeclareCustomizedJavaSerializer javaSerializerInfo shuffleGrouping EventHubSpout topologyBuilder SetBolt SQLBolt SQLStorageBolt Get new Dictionary string List 2 true shuffleGrouping TableStorageBolt Constants DEFAULT STREAM ID addConfigurations new Dictionary string string topology tick tuple freq secs 1 topologyBuilder SetTopologyConfig new Dictionary string string topology workers partitionCount ToString The two bolts also need to be added to the topology The tableStorageBolt is configured with a name a method which can be used to create instances and a description of its output In this case I m emitting the email address to the default stream The parallelism hint is also configured and the boolean flag which indicates this bolt supports acking Since this bolt sits behind the Java spout we also need to configure how the data of the spout can be deserialized The bolt is also configured to use shuffleGrouping meaning that the load will be spread evenly among all instances The SQLBolt is configured accordingly with the additional configuration to support time ticks After I created an HDInsight Storm cluster on Azure I was able to publish the topology from within Visual Studio The cluster itself was created in 15 minutes It took quite some time to find the right parallelism for this topology but with the code I ve posted I got these results after one hour Table storage is slow the 2 SQL bolts are able to keep up with the 256 table storage bolts After one hour my topology was able to process only 10 million records Even though I was able to send all 100 million events to the event hub in under two hours processing would take 5 times longer One of the causes for this is the test data The partition and row key for the table storage is based on the email address class BreachedAccount TableEntity public BreachedAccount public BreachedAccount System Net Mail MailAddress emailAddress string leakedIn this PartitionKey emailAddress Host this RowKey emailAddress Address public string BreachedIn get set Yet none of the email addresses in my test data share the same partition key If the data was more realistic I would be able to use batches In the original Adobe data breach 115 million addresses were part of 130 different domains Table storage allows you to send batches of 100 records to the same partition key which should result in less processing time I ll regenerate the test data with the same characteristics modify the table storage bolt and rerun the experiment at a later time hoping to get a better result The original standard instance of SQL Server was scaled to an S3 which allowed me to stay just below the max throughput with batches of 1000 records At the moment I actually have the same performance as Troy s setup but with quite a different approach Storm and HDInsight were new to me and I enjoyed the programming model The scale and possibilities it can give are enormous yet every part of your topology needs quite some attention if you want to have a high throughput solution The NET part of the SDK is not yet entirely complete so you can t just port Java snippets to C and expect them to work I also ran into some NotImplementedExceptions Microsoft is still busy implementing the SDK and the team is actively adding examples and functionality Further reading Event Hubs Programming Guide Storm Applied Introduction to Apache Storm on HDInsight Real time analytics for Hadoop HDInsight Storm Examples Process events from Azure Event Hubs with Storm on HDInsight Performance Tuning for HDInsight Storm and Microsoft Azure EventHub Understanding the Parallelism of a Storm Topology Author BennyM Posted on 08 06 2015 08 06 2015 Categories Software Development Tags azure event hub hdinsight storm 4 Comments on Using HDInsight Storm to process 100 million events

    Original URL path: http://blog.bennymichielsen.be/category/softwaredevelopment/ (2016-04-29)
    Open archived version from archive

  • IoT – Benny Michielsen
    so I was running the latest bits sudo apt get update sudo apt get upgrade Next up programming I briefly looked at the options I had I could program in C Python C and many others But time was limited this weekend I live and breath NET so I changed the list to NET Core or Mono I chose Mono because I had experimented with it years ago and NET Core has not yet reached a stable point Toying around with alpha and beta releases was not on my todo list for today The default package repository has a very old version of Mono so you need to follow the instructions on the Mono site Add the signing key and package repository to your system then run sudo apt get install mono runtime sudo apt key adv keyserver hkp keyserver ubuntu com 80 recv keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo deb http download mono project com repo debian wheezy main sudo tee etc apt sources list d mono xamarin list sudo apt get update sudo apt get install mono runtime I created a Hello World console application on my Windows 10 laptop and used PSFTP to copy the exe to the Pi It just worked Then the search was on to find a library to interface with the GPIO pins on the Pi After looking around I found Raspberry Sharp IO It had the API I wanted You can use the event model to track changes in the GPIO pins just what I needed var pin2Sensor ConnectorPin P1Pin11 Input GpioConnection connection new GpioConnection pin2Sensor connection PinStatusChanged sender statusArgs Console WriteLine Pin changed statusArgs Configuration Name Deploying this to the Pi however resulted in catastrophic failure Some weird error message pi raspberrypi ticktack sudo mono TickTackConsole exe Missing method ctor in assembly home pi ticktack Raspberry IO GeneralPurpose dll type System Runtime CompilerServices ExtensionAttribute Can t find custom attr constructor image home pi ticktack Raspberry IO GeneralPurpose dll mtoken 0x0a000014 Assertion at class c 5597 condition mono loader get last error not met Stacktrace Native stacktrace Debug info from gdb New LWP 1965 Thread debugging using libthread db enabled Using host libthread db library lib arm linux gnueabihf libthread db so 1 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 40 sysdeps unix sysv linux waitpid c No such file or directory Id Target Id Frame 2 Thread 0x769f3430 LWP 1965 mono 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 Thread 0x76f5e000 LWP 1961 mono 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 Thread 2 Thread 0x769f3430 LWP 1965 0 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 0x76e65af4 in new sem wait sem 0x3181a4 at

    Original URL path: http://blog.bennymichielsen.be/tag/iot/ (2016-04-29)
    Open archived version from archive

  • mono – Benny Michielsen
    Assertion at class c 5597 condition mono loader get last error not met Stacktrace Native stacktrace Debug info from gdb New LWP 1965 Thread debugging using libthread db enabled Using host libthread db library lib arm linux gnueabihf libthread db so 1 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 40 sysdeps unix sysv linux waitpid c No such file or directory Id Target Id Frame 2 Thread 0x769f3430 LWP 1965 mono 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 Thread 0x76f5e000 LWP 1961 mono 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 Thread 2 Thread 0x769f3430 LWP 1965 0 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 0x76e65af4 in new sem wait sem 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 69 2 0x00219f98 in mono sem wait 3 0x0019091c in Backtrace stopped previous frame identical to this frame corrupt stack Thread 1 Thread 0x76f5e000 LWP 1961 Cannot access memory at address 0x1 0 0x76e67ee8 in libc waitpid pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 1 0x000c0ba4 in Cannot access memory at address 0x1 Backtrace stopped previous frame identical to this frame corrupt stack Got a SIGABRT while executing native code This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application Fast forward 4 hours It turns out installing mono runtime doesn t quite get you all the bits you need You need to run the following command sudo apt get install libmono system core4 0 cil After that the application started again The button I had connected to the pins activated my code Yay With that my time for the weekend was all used up but I m now ready to be create my actual application To be continued MarchIsForMakers Author BennyM Posted on 14 03 2016 Categories Software Development Tags IoT mono Raspberry Pi 1 Comment on Getting up and running with Mono and Raspberry Pi 3 MVC3 and MonoDevelop Just a quick update to list the assemblies you need to copy to make MVC3 work with Mono and MonoDevelop On my Mac I opened a Visual Studio solution I deleted all the entity framework references from my MVC project won t use it anyway and then copied these assemblies to a local folder since they are not part of the Mono framework System Web Helpers System Web Mvc System Web Razor System Web WebPages Deployment System Web WebPages System Web WebPages Razor Fix the references in your project hit run and you re good to go Unfortunately you re not getting any intellisense yet Author BennyM Posted on

    Original URL path: http://blog.bennymichielsen.be/tag/mono/ (2016-04-29)
    Open archived version from archive

  • Raspberry Pi – Benny Michielsen
    so I was running the latest bits sudo apt get update sudo apt get upgrade Next up programming I briefly looked at the options I had I could program in C Python C and many others But time was limited this weekend I live and breath NET so I changed the list to NET Core or Mono I chose Mono because I had experimented with it years ago and NET Core has not yet reached a stable point Toying around with alpha and beta releases was not on my todo list for today The default package repository has a very old version of Mono so you need to follow the instructions on the Mono site Add the signing key and package repository to your system then run sudo apt get install mono runtime sudo apt key adv keyserver hkp keyserver ubuntu com 80 recv keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo deb http download mono project com repo debian wheezy main sudo tee etc apt sources list d mono xamarin list sudo apt get update sudo apt get install mono runtime I created a Hello World console application on my Windows 10 laptop and used PSFTP to copy the exe to the Pi It just worked Then the search was on to find a library to interface with the GPIO pins on the Pi After looking around I found Raspberry Sharp IO It had the API I wanted You can use the event model to track changes in the GPIO pins just what I needed var pin2Sensor ConnectorPin P1Pin11 Input GpioConnection connection new GpioConnection pin2Sensor connection PinStatusChanged sender statusArgs Console WriteLine Pin changed statusArgs Configuration Name Deploying this to the Pi however resulted in catastrophic failure Some weird error message pi raspberrypi ticktack sudo mono TickTackConsole exe Missing method ctor in assembly home pi ticktack Raspberry IO GeneralPurpose dll type System Runtime CompilerServices ExtensionAttribute Can t find custom attr constructor image home pi ticktack Raspberry IO GeneralPurpose dll mtoken 0x0a000014 Assertion at class c 5597 condition mono loader get last error not met Stacktrace Native stacktrace Debug info from gdb New LWP 1965 Thread debugging using libthread db enabled Using host libthread db library lib arm linux gnueabihf libthread db so 1 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 40 sysdeps unix sysv linux waitpid c No such file or directory Id Target Id Frame 2 Thread 0x769f3430 LWP 1965 mono 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 Thread 0x76f5e000 LWP 1961 mono 0x76e67ee8 in libc waitpid Cannot access memory at address 0x1 pid 1966 stat loc 0x7e904960 options 0 at sysdeps unix sysv linux waitpid c 40 Thread 2 Thread 0x769f3430 LWP 1965 0 0x76e65a40 in do futex wait isem isem entry 0x3181a4 at nptl sysdeps unix sysv linux sem wait c 48 1 0x76e65af4 in new sem wait sem 0x3181a4 at

    Original URL path: http://blog.bennymichielsen.be/tag/raspberry-pi/ (2016-04-29)
    Open archived version from archive

  • Running Gulp and NPM in VSO Build fails with “Run as Administrator” message – Benny Michielsen
    and suddenly the issue was resolved Since I wanted to write this down I had to reproduce the issue So I removed the attribute again but the build somehow kept building I had also deleted the failed builds so I have no log anymore to illustrate the issue Something has changed though as the builds now take much longer than before even without the force attribute You can see that in the image below Hopefully I don t run into it again I added the force attribute after reading this GitHub issue Author BennyM Posted on 29 02 2016 29 02 2016 Categories Software Development Tags build vso Leave a Reply Cancel reply Your email address will not be published Required fields are marked Comment Name Email Website Post navigation Previous Previous post Session Storm with HDInsight Next Next post Getting up and running with Mono and Raspberry Pi 3 Search Search for Search Follow me Recent Comments Rick on Partitioning and wildcards in an Azure Data Factory pipeline BennyM on WCF HTTPS And Request Entity Too Large Justin on WCF HTTPS And Request Entity Too Large The Morning Brew Chris Alcock The Morning Brew 2052 on Getting up and

    Original URL path: http://blog.bennymichielsen.be/2016/02/29/gulp-npm-vso/ (2016-04-29)
    Open archived version from archive